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(57) Abstract: The invention 
relates to methods, systems, and 
programming for displaying media 
in a scaled manner and/or in an 
orientation other than an operating 
system displays its graphical user 
interface. Some embodiments 
lay out digital content at a viitual 
pixel resolution and then display 
a portion of the layout at a smaller 
display resolution by displaying 
images and text at scaled-down 
coordinates and scaled-down 
sizes, with the display of text 
being composed from a plurality 
of font bitmaps having character 
shapes and pixel alignments 
selected to improve readability 
at the scaled-down size. Some 
embodiments, which operate 
on computing systems having 
operating systems that display 

^ — ■ " ■' ^ a graphical user interface in a 

portrait orientation, display a scaled-down representation of a web page in a landscape orientation, including displaying in such 
a landscape orientation scaled-down images and scaled-down text composed from a plurality of font bitmaps designed for the 
scaled-down size. 
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METHODS AND SYSTEMS FOR DISPLAYING MEDIA IN SCALED MANNER AND/OR IN ORIENTATION 

FIELD OF THE INVENTION 

The present invention relates to methods, systems, and programming for displaying media 
In a scaled manner and/or in an orientation other than a computer s^tem's operating system 
displays a graphical user interface. 

BACKGROUND OF THE INVENTION 

This patent application has many aspects which relate to the optimization of using 
computing devices with small or low resolution screens, such as handheld computers, cellphone 
computersm, or computers with wrist or head mounted displays. A good portion of this optimizing 
has been done to improve the use of such small screen devices for browsing the World Wide Web 
or similar media, although many of its features can be used when viewing other types of screen 
content. 

Another portion of this optimization has been focused on improving the ability to browse 
such media through a relatively low bandwidth links, such as those that would be found on cun^nt 
wireless linlo. It should be appreciated, however, that many aspects of the inventions disclosed in 
this application are not limited to use for these purpose. 

For example, some of the features which are designed to make it easier for users to view 
portions of Web pages at a larger size could be used to make reading the Web on traditional 
computers easier on the eye or easier to read at greater distances. 

-At the time this application Is being filed there are multiple handheld computers which 
have approximately 240 by 320 pixel screens which measure approximately four Inches diagonally. 
These include the Compaq iPaq H3650 Pocket PC, the Casio Cassiopeia, and the Hewlett- 
Packard Jomado 525. Unfortunately such a resolution would be too low to display most cunrent 
Web pages on. Currently most Web pages can be viewed v\rith 640 x 480 resolution screen 



wo 02/088979 



PCT/US02/14380 



-2- 

(although a few web sites cannot even be properly viewed at ttiis resolution), it would be desirable 
to be able to view most web pages with such hand heid devices. 

-The manufacturers of iiquld crystal displays are now capable of making screens having 
substantially higher resolutions than those which are cunentty on the market. Makers of organic 
LED displays daim they can achieve even higher resolutions. This means that a four inch diagonal 
screen of the size cunently in the handheld computers listed above could have a resolution of 480 
by 640 or higher. Although such screens would provide an acceptable resolution for many web 
sites, even a higher effective resolution would be desirable to view many web pages. 

In additton, in order for such screens to be seen at a relatively high resolution, they would 
have to be held close to a user's eyes. Although this might be satisfactory for many applications, 
users might often find it tiring or inconvenient to constantly hold a handheld computer close one 
eyes. 

-Such advances in display resolution would also mean that a 320 by 240 pbcel screens 
could be made with a diagonal length of two inches or less. Such a display would be about the 
size of the display commonly contained in many present-day cellphones, and could also fit onto a 
wristwatdi. Such displays wouki make many forms of applications cunrently used on hand held 
computers available on cellphones, wristwatches or other similarly small fonnat computers. 
Unfortunately they would have the problem of both having a relatively low resolution which would 
tend to make it difficult for them to view most web pages, and of being so physically small that for a 
user to be able to see their resolution they would have to be held very dose to the user's eyes. 
Again, holding such a device dose to a user's eyes might be very satisfactory at periods of time, 
but over long periods of time, or In certain situations it might be inconvenient. 

-Cunently there are several companies which provide head mounted displays vrfiich 
enable a person to see an image of a computer screen, either as a result of light reflected into the 
user's eyes through a device that appears somewhat like a pair of glasses, or from a minor placed 
above, below, or off to the side of the user's eyes. To make it easy for user to Interact with their 
sunxjundings while using such a head mounted display, it is often desirable to have such projected 
computer screens take up a relatively small portion of the user's optic field. Thus, users of such 
displays might face many of the same problems as would users of small handheld screens. 
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-Some aspects of the Invention relates to methods for optimizing the browsing of the Web 
or application screen output on a computer with relatively limited computational power, memory, or 
bandwidth to the Internet. For example, cunently a standard Web browser of the type used In 
most desktop and laptop computers requires many megabytes of memory and a relatively large 
amount of computational power. They also require a connection to the Intemet having at least the 
speed of a high-speed modem to work effectively with a type of Web content contained in many of 
the Worid Wide Web's more frequently used Web pages. Unfortunately, many handheld 
computers either do not have the storage or computational capacity to be dble to effectively view 
many such web pages. Also most commonly available wireless systems have a bandwidth which 
Is substantially below that which would be desirable for viewing many Web pages. As a result of 
these factors, one of the focus is of some of the Innovations contained in this application relate to 
methods for enabling computers with limited storage, commutation, or bandwidth to better browser 
the Worid Wide Web or similar media. 



SUMMARY OF THE INVENTIQN 

The invention is most accurately described by the claims below and where the claims differ 
from this summary, the wording of the claims should be considered the true description of the 
invention. 

One asped of ttie invention relates to displaying digital content by accessing it, laying it 
out at a virtual pbcel resolution using layout sizes for image and fonts and displaying at least a 
portion of tiie layout at a smaller display resolution. This includes displaying ttie images and text at 
coordinates and sizes scaled down from ttieir layout size by tiie scale feotor. it also involves 
displaying text by composing images of strings from separate font bitmaps having character 
shapes and pbcel alignments that have been selected to improve readability at the scaled down 

Si2B. 

Anottier aspect of ttie invention relates to accessing a web page. Then - on a screen of a 
computing system witti an operating system tiiat displays a graphical user interface in a portrelt 
orientation - displaying a scaled-down representation of ttie web page In a landscape orientation. 
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This includes displaying on the screen, in a landscape orientation, images that are scaled-down 
and text composed of fonts bitmaps displayed In the landscape orientation. 



BRIEF DESCRIPTION OF THE DRAWINGS 

These and other aspects of the present invention will become more evident upon reading 
the following description of the preferred embodiment in conjunction with the accompanying 
drawings, in which: 

FIG. 1 illustrates a process used according to some aspects of the present invention to 
improve Web browsing and/or display of other tj^^es of computer generated content, particulariy on 
systems with relatively towHPesolution screens. 

FIG. 2 illustrates a networiced computing environment in which the invention can operate 
that Includes a portable browser, a proxy server, a Web server, and a font server. 

FIG. 3 illustrates an alternative networked computing environment in which the invention 
can operate that includes a browser and a Web server. 

FIG. 4 illustrates a second altemative networiced computing environment in which the 
Inventon can operate that Includes a browser and a Web server. 

FIG. 5 illustrates a third attemative networiced computing environment In which the 
invention can operate that includes a browser and a Web server. 

FIG. 6 illustrates a computer system in which the invention can operate that contains the 
standard Web content to be displayed and browser functionality containing a process for scaling 
and/or subpixel optimizing the content 
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FIG. 7 illustrates an alternative confiputer system in which the invention can operate that 
contains the content to be displayed, a proxy process for scaling and/or subpixel optimizing the 
content and browser functionality. 

FIG. 8 illustrates a second alternative computer system in which tlie invention can operate 
that contains previously scaled and/or subpixel-optimized content 

FIG. 9 illustrates a known vertically striped RGB LCD display device. 

FIG. 10 illustrates some of the aspects of the invention involved in performing the subpixel 
optimization of both images and text refen^ed to with regards to steps 108 and 112 of FIG. 1, 
respectively. 

FIG. 11 illustrates the level of readability provided by a cun-ent embodiment of the 
invention when displaying standard Web content on a 320 by 240 color display. 

FIG. 12 Illustrates the mapping of a pixel and subpixel grid over a portion of source image 

102. 

FIG. 13 is an expansion of a section of the mapping grid of FIG. 12. 

FIG. 14 Illustrates the positioning of a window over the source image used to calculate the 
luminosity of a red (R) subpixel of the lower resolution display device. 

FIG. 15 illustrates the positioning of such a window used to calculate the iunninosity of a 
green (G) subpbcel of the lower resolution display device. 

FIG. 16 illustrates the posttibning of such a window used to calculate the luminosity of a 
blue (B) subpixel of the lower resolution display device. 

FIG. 17 illustrates scan lines used In a scan line coverage method to calculate the subpbcel 
luminosity of a red subpixel by estimating the portion of the red subpixel's associated source image 
window that Is covered by one or more pbcel of different colors. 
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FIG. 18 illustrates similar scan lines used to calculate the luminosity of a green (G) 
subpixel of a lower resolution display device. 

FIG. 19 illustrates similar scan lines used to calculate the luminosity of a blue (B) subpixel 
of a lower resolution display device. 

FIG. 20 is a repeat of FIG. 17. 

FIG. 21 illustrates the portions of the horizontal scan line shown In FIG. 20 that are 
covered by different source Image pixels vrithin the red pixel's source image window. 

FIG. 22 illustrates the porhons of the vertical scan line shown in FIG. 20 that are covered 
by different source Image pixels within the red pixel's source image window. 

FIG.23isidenticaltoFiG.18. 

FIG. 24 illustrates the portions of the horizontal scan line shown in FIG. 23 that ara 
covered by different source image pixels within the green pbcel's source image window. 

FIG. 25 Illustrates the portions of the vertical scan line shown in FIG. 23 that are covered 
by different source Image pixels within ffie green pixefs source image window. 

FIG. 26 is Identical to FIG. 19. 

FIG. 27 illustrates the portions of the horizontal scan line shown in FIG. 26 that are 
covered by different source image pixels within the blue pixel's source image window. 

FIG. 28 illustrates the portions of the vertical scan line shown in FIG. 26 that are covered 
by different source image pixels within the blue pixel's source image window. 
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FIG. 29 is a highly simplified pseudocode description of a subpixel optlnfiization metliod 
which calculates subpixel luminance values based on line coverage values, such as the line 
coverage values illustrated with regard to FIGS.17 through 28. 

FIG. 30 illustrates how two horizontal and two vertical scan lines can be used on altemate 
embodiments of "line coverage" methods for calculating the colors of pixels in subpixei-optimized 
scaled images. 

FIG. 31 illustrates how two diagonal scan lines can be used on altemate embodiments of 
line coverage" methods for calculating the colors of pixels in subpbcel-optimized scaled images. 

FIG. 32 illustrates how a combination of two diagonal, two horizontal, and two vertical scan 
lines can be used on alternate embodiments of 'line coverage" methods for calculating the colors 
of pbcels in subpbcel-optimized scaled images. 

FIG. 33 illustrates line coverage for two horizontal coverage lines at a 1/2 horizontal and 
vertical scaling. 

FIG. 34 illustrates line coverage for two vertical coverage lines at a 1/2 horizontal and 
vertical scaling. 

FIG. 35 illustrates line coverage for two horizontal coverage lines at approximatel/ a 215 
horizontal and vertical scaling. 

FIG. 36 illustrates line coverage for two vertical coverage Ihes at approximately a 215 
horizontal and vertical scaling. 

FIG. 37 illustrates line coverage for two horizontal coverage lines at approximately a 2/3 
horizontal and vertical scaling. 

FIG, 38 illustrates One coverage for two vertical coverage lines at approximately a 2/3 
horizontal and vertical scaling. 
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FIG. 39 (llustrates the source (mage pixel areas used in an "area coverage' method of 
c^culating the color values of a subpbcel-optimized scaled image. 

FIG. 40 is similar to FIG. 39 except that it uses different hatching to illustiBte the areas of 
different source image pixels within a source Image window used to calculate a subpixel's 
luminosity value according to one such "area coverage" method. 

FIG. 41 IS a highly simplified pseudocode description of a subpixel optimization method 
which calculates subpixel luminance values based on aread coverage values, such as those 
discussed with regard to FIGS 39 and 40. 

FIG. 42 illustrates a method of producing a scaled subpixeKoptimized Image of a bitmap 
image by associating a luminosity value with a red subpixel. 

FIG. 43 illustrates a mefiiod of producing a scaled subpixel-optimized image of a bitmap 
image by associating a luminosity value with a green subpixel. 

FIG. 44 illustrates a method of producing a scaled subpixel-optimized Image of a bitmap 
image by associating a luminosity value with a blue subpixel. 

FIG.FIG. 45 illusliates a set of gray scale pbcels under an RGB grid. 

FIG. 46 illustrates how the luminosity associated with an individual pixel is distribution 
under a traditional linear filtering method. 

FIG. 47 illustrates the resulting subpixel luminosity values as a result of the traditional 
linear filtering method. 

FIG. 43 illustrates the distribution of the minimum source image luminosity values under a 
non-linear filtering. 

FIG. 49 Illustrates the distribution of the excess luminosity values under a non-linear 
filtering method. 
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FIGS. 50 through 52 compare the results of the linear and non-linear filtering methods with 
the original source pixel luminosities. 

FIG. 53 is a highly simplified pseudocode representaton of a software method for creating 
a subplxel-optlmized representation of a bicoior bitmap. 

FIG. 54 Is a flow diagram of a process to allow a user to dynamically tradeoff color and 
positional resolution. 

FIG. 55 illustrates the mapping between a character-font shape defined by an outline font 
description and an array of pbcels having subpixels used to represent that shape on a subpixel 
addressable display; 

FIG. 56 is a screen shot of a 320 x 240 screen of a web page produced by one 
embodiment of the present Invention; 

FIG. 57 is a 2x blowup of the screen shot shown In FIG. 56; 

FIG. 58 Illustrates how a computer can access font bitmaps or font outlines from a font 

server; 

FIG. 59 illustrates how a computer can access font bitmaps or font outlines which it has 
stored within it; 

FIG. 60 is a highly simplified pseudocode representation of an algorithm for calculating a 
subpixel-optimized bitmap of a character-font shape using non-linear color balancing of the type 
illustrated in FIGS. 48 through 52; 

FIG. 61 through 63 illustrate the size of the virindow In a character-font shape image, such 
as that shown in FIG. 55, used to detemiine a coverage value for each of the three subpucePs of an 
individual pbcel of a pbcel array such as that shown in FIG. 55; 
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FIG. 64 through 67 illustrates some prior art techniques which have been used to calculate 
coverage values for non-square rasterization units (usually whole pixels in the prior art); 

FIG. 68 through 87 Illustrate a computationally efficient method of calculating the coverage 
value of rasterizafion units, using weighted line coverage valueSi which method is used In some 
embodiments of the present invention to calculate a coverage value for subpbcels; 

FIGS. 88 through 90 illustrates some of the other anangements of coverage lines which 
can be used vioth a weighted line coverage algorithm of the general type described above with 
regard FIG. 68 through 87; 

FIG. 91 illustrates a mapping of an anay of pixels, and their respective subpixels, into an 
image of a portion of a font outline; 

FIG. 92 illustrates corresponding coverage values which have been calculated for the 
subpixel's of the anay shown in FIG. 91; 

FIG. 93 corresponds to FIG. 46, and like RG. 46 illustrates how a prior art linear color 
balancing method distributes all of an individual subpixeFs coverage value over a series of adjacent 
subpixeFs within a given pixel row; 

FIGS. 94 and 95 illustrates color balance filters which can be used with the non-linear color 
balancing method described with regard FIG. 60; 

FIG. 96 is a highly simplified pseudo code description of an algorithm which can be used to 
represent the whole pixel alpha values calculated for font bitmaps by a method such as that 
described with regard FIG. 60 into a more limited Color Space of such whole pixel alpha values; 

FIG. 97 Is a highly simplified pseudo code description of an algorithm for displaying text 
strings on a subpixel addressable display using font bitmaps created by a combination of the 
methods Illustrated In FIG. 60 and 96; 
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FIGS. 98 through 101 illustrate how well the present invention can display web pages on a 
320 X 240 screen, with FIGS. 98 and 100 each being a screen shot of a 640 x 480 layout of a 
different web page, and FIGS. 99 and 101 showing how the present invention is capable of 
displaying each of these two web page is on a 320 x 240 screen; 

FIG. 102 is a schematic block diagram of some of the data structures and programming 
used by a proxy server and thin client computer to enable a user of the thin client computer to 
access web content on a scaled-down, subpixel-optimlzed screen; 

FIG. 103 is a portion of the IHTMLcode of the web page Illustrated in FIGS. 98 and 99; 

FIG. 104 illustrates the layout of a web page produced by the proxy server and the portion 
of that layout which falls within the proxy server's virtual screen, which In this example coniesponds 
to the portion of the web page shown bi FIG. 99; 

FIGS. 105A and 105B are highly simplified pseudo code descriptions of programming on 
the proxy server shown in FIG. 102; 

FIGS. 106A through 106C are highly simplified pseudo code descriptions of programming 
for capturing and downloading a scaled-down and subpbcel-optimized representation of a portion of 
a web page to a thin client computer; 

FIG. 107 is a highly simplified pseudocode description of programming for the actual 
downloading of the web page representation captured by the programming of FIGS. 106A through 
106C; 

FIG. 108 is a highly simplified representation of the data downloaded to a thin client 
computer by the programming illustrated in FIG. 107; 

FIGS. 109A through 109C are highly simplified pseudocode representations of 
programming on the thin client shown in FIG. 102; 
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FIGS. 1 10 through 1 12 illustrate how, if a user clicks on a text entry field on a web page's 
display on the thin client shown in FIG. 102, a pop-up menu is shown that allows the user to enter 
text into that field; 

FIG. 113 illustrates how the user can use the same pop-up keyboard to enter URLs which 
allow him or her to access desired websites; 

FIG. 114 is virtually Identical to FIG. 13 except that it illustrates an embodiment of the 
invention having a toolbar at the top of the thin client computer screen which includes graphteal 
user interface buttons and a URL text entry field; 

FIGS. 115 and 116 are h^hly simplified pseudocode representations of programming on a 
proxy browser and thin client computer, respectively, which Is used In an altemate embodiment of 
the invention in which the thin client computer stores the layout of an entire web page to allow it to 
scroll and zoom relative to that web page at a higher speed; 

FIG. 117 is a schematic illustration used to help explain the operation of the pseudocode 
illustratedinFIGS.115and116; 

FIGS. 118 through 120 illustrate how the present invention can allow a user to rapidly 
select a portion of a web page's or a screen's contents and tiien zoom to that selected portion; 

FIGS. 121 ttirough 128 illustrate a zoom-click aspect of tiie invention which allows a user 
to view and select portions of a web page or a screen with greater accuracy; 

FIG. 129 is a highly simplified pseudocode description of programming for allowing a user 
to have selected text reflown across a given screen widtii at a larger scale; 

FIGS. 130 tiirough 137 am used to illustrate how tfie text reflow capability shown in FIG. 
129 can operate; 

FIG. 138 is used to Illustrates how multiple client computers can be programmed to access 
a common font server and/or a common proxy server; 
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FIG. 139 is a highly simplified pseudocode description of programming which can be used 
on one or more font servers; 

FIG. 140 illustrates how certain aspects of the present Invention can be used to allow a 
client computer to view screens which are drawn by applications (which can inciudei but are not 
limited to, one or more web browsers) ninning on a remote computer in a scaled-down, subpixel- 
optimized manner by Intercepting calls made by such applications to the remote computer's 
operattng system; 

FIG. 141 illustrates how subpixel-optimtzed, scaled-down views can be had of screen 
output generated by application programs (which can include but are not limited to one or more 
web browsers) mnning on a given computer, even if those applications have not been programmed 
to generate such views, on the screen of that given computer, by intercepting calls to the 
computer's operating system made by such applications; 

FIG. 142 illustrates how certain aspects of the present invention can be used allow 
portable small-screen, thin-client computers to access web content and the screen output of 
various application programs through both local and/or internet wireless communication; 

FIGS. 143 and 144 are used to illustrate how in some embodiments of the present 
invention subpixel-optimized output is displayed with a landscape orientation by rotating a 
computing device that has an operating system programmed to work in a portrait orientation; 

FIG. 145 is a highly simplified pseudocode description of programming use to draw a 
simple shape with a subpixel-optimized resolution; 

FIG. 146 is a highly simplified pseudocode description of how web applets can be used to 
draw subpixelK)ptimized elements on the screen of acomputen 

FIG. 147 is a highly simplified blocic diagram iiiustrating how rollover images can be 
subpixel-optimized; 
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FIG. 148 is a highly simplified block diagram iliustrating how GIFF animafions can be 
subpixel-optimized; 

FIG. 149 is a highly simplified pseudocode description of how 3-D animation can be 
subpixel-optimized; 

FIGS. 150 and 151 are highly simplified pseudo code descripttons of how a client/server 
gaming system can be used to provide subpixekptimized game images on a client computer; 

FIG. 152 is a highly simplified pseudocode description of how subpixel-optimized displays 
can be made of images having con^esponding transparency maps; 

Figure 153 is a highly simplified pseudocode description of how video using interpolation 
between keyframes can be subpixel-optimized; 

FIG. 154 is a highly simplified pseudocode description of how video whose representation 
includes the drawing of screen changes to less than a whole frames can be subpbcel-optimized; 

FIGS. 155 and 156 are highly simplified pseudocode description of different methods of 
displaying images which move relative to a display window; 

FIGS. 157 through 159 are highly simplified pseudocode description of how subpixei 
optimization can be applied to video which is been represented by various compression 
techniques; 

FIG. 160 is a highly simplified pseudocode representation of programming for enabling a 
server computer to download subpixel-optimized, scaled-down video to a client computer, 

FIG. 161 is a highly simplified pseudocode description of programming on both a client and 
proxy computer to enable the client computer to access scaled-down, subpixel-optimized video 
from other servers through a proxy computer; 
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FIG. 162 is a highly simplified pseudocode representation of programnning which allows 
electronic infc to be viewed nnofe cleariy; 

FIGS. 163 through 166 are used to help illustrate the benefits of the programming describe 
with regard FIG. 162; 

FIG. 167 is used to illustrate that the present Invention relates not only to methods, but 
also to programming and data related to such methods stored in a machine readable form or 
embodied in a propagated signal, and to programmed and/or hardwired computer systems for 
performing such methods and/or use such programming and/or data. 

H6S. 168 through 184 are used to describe additional improvements to the invention for 
improving the clarity of subpbcel-optimized color-balanced font bitmaps produced by the present 
Invention. 

FIG. 185 is a higher level description of the selected-text reflow method described with 
regard to FIGS. 129 through 134; 

FIG. 186 is a high-level pseudocode description of a zoom-to-fit method;, of the general 
type described with regard to FIGS. 118 through 120; 

FIG. 187 is a high-level pseudocode description of a drag scroll method; that allows a user 
to easily navigate within the display of a web page's layout; 

FIG. 188 is a high-level pseudocode description of a ciicl<-zoom method that enables the 
user to rapidly selected to zoom in on a desired portion of the display of a layout of a web page; 

FIG. 189 is a highly simplified pseudocode description of the zoomcllcic method described 
in some detail with regard to FIGS. 121 through 128; 

FIG. 190 is a highly simplified pseudocode description of a method that allows a user to 
see a zoom-out view of a web page using greeldng; 
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DETAILED DESCRIPTION OF SOME PREFERRED EMBODIMENTS 

FIG. 1 Is a high level diagram that represents basic processes and data representations 
that may be used according to some aspects of the present invention to improve Web brovtfsing 
and/or display of other types of computer generated content, particularly on systems with low 
resolution displays. 

Digital content 100, Including one or more bitmap images 102 and text 104 is displayed In 
a subpixel-optimized downscaled fonnat 106. In one embodiment of the invention a difference 
process, comprised of step 108 Is used to subpixel-optimize the display of the bitmap images 102. 
than Is used to subpixel optimize the display of the text content. Step 108 uses a subpixel 
opUmtzation routine that is particularly suited for producing subpixel-optimized images from color 
bitmaps. The process 108 also scales down the bitmaps for display on screens having a lower 
resolution than that at which most Web content is currently displayed. 

The text 104 contained in the digital content 100 is processed for display on a small 
resolution subpixel addressable screen by using steps 110 and 112. Step 110 replaces the fonts 
normally used to display text with fonts that are optimized for display at small resolutions on 
subpixel-optimized screens. Then step 1 12 uses font bitmaps from the substitufed fonts that have 
been produced by a subpixel optimization roufine particulaily suited for the representation of high 
resolution Images of shapes of a unifomi color, such as the mathematically defined outlines 
commonly used to define font shapes. 

One use of the present invention is in the context of a portable, low resolution Web 
browser that displays images and/or text, represented by a marlcup language, that have been 
downloaded from Oie Internet 

To date there have been multiple so-called mark-up languages. One of the eadiest and 
msi successful was SGML (Standard General Mariaip Language). SGML is a text-based 
language that can be used to 'maricup' data vrith descriptive 'metadata' that provides Information 
about the data. As an e)Qimple, markup metadata can be used to indk^ate the purpose for which 



wo 02/088979 



PCT/US02/14380 



-17- 

the data is Intended or the location on a Web page at which the data should be positioned. It can 
also be used to Indicate links to data of other types, such as Images, which are to be Inserted at a 
given location in a text, or in a document described by the mark-up language. Several markup 
languages that are commonly used today, such as HTML and XML, are derived from SGML 

In a preferred embodiment of the present invention the digital content 100 referred to in 
FIG. 1 above may be standard Web content that includes text and/or images represented by a 
markup language such as HTML. This standard Web content 100, representing perhaps a Web 
site home page, can be downloaded through various apparatus and methods described below for 
display on a portable low resolutton browser device 200, shown in FIGS. 2 through 4. Before 
display on the browser devtee 200 the digital content 100 may be scaled and/or subpixel-optlmized 
for enhanced readability tiirough various methods and processes, such as those described below. 

FIG. 2 Illustrates a networked computer environment implemented in accordance with one 
embodiment of the present Invention. The thin client browser 200 program runs in a handheld or 
other small computing devtee capable of retrieving and displaying text and/or graphics on a small 
liquid crystal display (LCD) screen. The browser allows a user to request digital infonnation from a 
remote source, e.g., from the Intemet and to display it on a screen. 

In the embodiment of the present invention illustrated in FIG. 2, a user would request the 
retrieval and display of digital content, containing Images and/or text, by way of manipulation of the 
controls of the thin client browser 200. The requested digital content may be a specific Web page 
accessible over the Internet The thin client browser 200 then makes its request 202 for content 
through a physically remote proxy server 210 over a networi^ 138, which can be, for example, a 
UVN, a WAN, or the internet 

The proxy server 210 mns a proxy process 216 that responds to the request for digltei 
content by generating a corresponding request 214 to a physically remote Web server 220 that 
contains the digitel content 100 requested by the user Server 220 responds to the proxy server 
request 214 by a download 222 of the digitel content 100 over the networic 138 to the proxy server 
210. 
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The proxy process 216 within the proxy server 210 then uses its computational resources 
to scale and subplxel optimize the digital content 100 according to steps 108, 110, and 112 
illustrated in FIG. 1. Scaling and subpixel optimizing are aspects of the present invention that 
result in the enhanced readability of images, such as text and/or graphics, on liquid crystal display 
devices. They will be discussed in greater detail in a subsequent section. 

The proxy server 210 completes a download 212 of the now scaled and subpbcekptimized 
content to the browser 200. At this point the user is able to view the content on the screen of the 
browser 200. 

In the embodiment of the invention shown in FIG. 2, the text portion of the digital content Is 
downloaded to the browser in the fomi of one or more strings of characters and designation of the 
font family, font size, and other font attribute. If the thin browser does not have bitmaps for any 
character in such a string in the font size and family specified for it, it requests one or more such 
bitmaps from a font server 230. In various embodiments of the inventions shown in FIGS. 2 
through 8, the proxy server could provide such font bitmaps or the thin client could have them as a 
standard part of Its software (although that would increase the size of the browser software). In still 
other embodiments the fonts could be outline fonts. The advantage of font bitmaps is that some 
font vendors are more willing to allow bitmaps of their fonts to be distributed more freely than 
outlines of such fonte. 

An alternate embodiment of the present invention is illustrated in FIG. 3. In tiiis 
embodiment ttie proxy server 210 and the Web server 220 of FIG. 2 are replaced with a single 
remote server 220A. The tiiin client browser 200 makes its request 202A for digital content 100 to 
the remote server 220A over a network 138. For example, networic 138 may betiie Internet or a 
LAN and the digital content 100 may be a specific Web page. The remote server 220A contains 
the requested digital content 100 and nins a proxy process 216A tiiat responds to the request 
202A. This process proxy process 21 6A can be any process mnning on the sender which 
dynamically scales and/or subpixel optimizes web content for display on the ttiin client browsers. 
The proxy process 21 6A operates upon ttie stored digital content 100 and dynamically converts it 
to tiie fonn 106 shown in FIG. 1 by perfbnning tiie steps 108, 110, and 112. The remote server 
220A completes a download 212 of ttie scaled and/or subpixei-optimized content to the thin client 
browser 200. 
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Another alternate embodiment of the invention is iliustrated in FiG. 4. As in FIG. 3 the thin 
clients request Is made directly to a remote server, in this case server 220B, over a networi( 138. 
In this implementation the remote server 220B contains the requested digital content In both a 
standard forni 100, which is for use by standard browsers computers, and the scaled and/or 
subplxel-optimized content 100A. The conversion from standard digital content 100 to the scaled 
and/or subplxel-optimized torn has occuned In advance thereby eliminating the need for a proxy 
process to dynamically convert it The thin client provides infonnation to the server indicating that 
It should receive the scaled and/or subplxel-optimized versbn of the requested content. The 
remote server 220B completes a download 212 of the scaled and/or subplxel-optimized content 
100 to the thin client browser 200. 

A further alternate embodiment Is illustrated In FIG. 5. Browser 200A is a full scale 
browser which also contains a scaling and/or subpixel optimization process 510. The browser 
200A makes a request 202B to a remote server 220C over the network 138 for the digital content 
100. Server 220C completes a download 212A of the requested digital content 100 to the browser 
200A. The conversion of the digital content 100 to a scaled and/or subplxel-optimized fbnn is 
handled by a process 510 running in the browser 200A. 

FIG. 6 illustrates a single computer system 600 capable of scaling and/or subpixel 
optimizing digital content 100. In this prefened embodiment the digital content has been created 
on or loaded into computer system 600 In advance. Computer system 600 contains a browser 
process 620 that Includes a scaling and/or subpixel optimizing sub-process 640. Here the user 
makes a request to the computer system 600 by way of an attached input device, e.g.. a keyboard 
or mouse, for the display of the digital content 100. The browser process 620 retrieves the 
requested digital content 100 from one of the computer system's storage elements, e.g., electronic 
memory or disk storage. Once retrieved the browser process 620 then passes the digital content 
to the scaling and/or subpixel optimizing sub-process 640. Once the conversion is complete, the 
converted content Is displayed on the display screen of the computer system 600. This 
embodiment of the present Invention operates witiiout tiie need for a network or remote servers. 

FIG. 7 illustrates an alternate single computer system implementation. In tills 
embodiment, computer system 700 contains ttie digital content 100 (e.g., tfie contents of a specific 
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Web page) that has been created or loaded in advance, a proxy process 740, and a browser 
process 720. The proxy process 740 executes scaling and/or subpixel opUnuzation programming 
760. The browser process passes a user request for display of the digital content 100 to the proxy 
process 740. Proxy process 740 then retrieves the digital content 100 from the storage element of 
computer system 700. Once retrieved, the programming 760 converts the digital content 100 to a 
scaled and/or subpixel-optimized fornn which is then passed to the browser process 740 for display 
by the display de\rice of computer system 700. 

FIG. 8 illustrates a second alternate single computer system Implementation. Here 
computer system 800 contains scaled and/or subpixekptimlzed Web content 810. A browser 
process 820 handles user requests for display of the content 810, retrieves it from the storage 
elements of computer system 800, and displays it on the screen of computer system 800. 

in the embodiments of the present invention presented above, the scaling of the image 
from the resolution of the source image to ttie resolution of the subpixel addressed screen has 
been detemiined, in part, by the respective resolutions of the source Image and the subpixel 
addressed display screen. 

In some embodiments of the invention, tiie determination of the scaling factor between the 
source image resolution and the resolution to be displayed on the subpixel addressable display 
screen can be specified by the user of the browser device. In this embodiment the user of the 
browser selects from a plurality of scale factors by communicating the s(^Ie factor to the process 
that scales down an image read from storage. The process that scales down the image read from 
storage then scales down and subpixel optimizes the image by a horizontal and vertical scale 
factor that varies as a function of the selected scale factor. 

As with most other user inputs to the browser device, such scale selections can be made 
by use of physical or GUI buttons, menu items, dialog boxes, or any other known user interface 
device on the browser device. 

In some such embodiments the user of the browser device may choose a second scaling 
factor from a plurality of scaling factors, according to which the digital content will be re-scaled and 
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re-subpixel optimized and redisplayed, after tl)e image has been previously retrieved from storage 
and displayed in subpixek>ptimized form at a first scaling f^tor. 

In such embodiments, the scaling fector used In the first scaled and subpixel-optlmized 
display may have been as a result of a default or prefened scaling factor or it may have been as a 
result of a scaling factor previously chosen by the user of the brower device. The user of the 
browser device may choose Irom a plurality of scaling factors for the redisplay of the digital content 
by the method of manipulating the control apparatus of the btomer device. Such manipulation of 
the control apparatus of the browser device will cause the image to be scaled according to the 
second chosen scaling factor. 

Such a second scaling may occur as a result of a process running either vAMn the 
browser device or within a physically remote server. The user of the browser device may continue 
to select from a plurality of scaling factors for subsequent redisplays. 

It is easiest to downscale digital Images by integer multiples, which cause an Integer 
number of pixels In a source inmge to fit into a given pixel In the rosulting downscaled image. For 
example, the scaling from a 640 by 480 resolution to a 320 by 240 resolution is a downscaling by a 
factor of two. Some embodiments of the present invention allow the user to select from a plurality 
of downscale fectors, Including non-Integer downscaling Victors. An example of a non-Integer 
downscaling factor is that of a 3/2 downscaling factor that would cause a 480 by 360 pixel portion 
of a 640 by 480 resolution source image to be scaled and/or subpbcei optimized for display on a 
320 by 240 resolution display screen. 

Computer graphic displays such as cathode ray tubes (CRT) or liquid crystal display (LCD) 
screens almost exclusively use the RGB model of color space, although the invention can be used 
with other color models, such as the CMYK color model. In the RGB model, the ttiree primary 
additive colors, red, green, and blue, are blended to fonn a desired color as perceived by the 
human eye. 

Most portable computing or Imaging devices have LCD screens that use the RGB model. 
Such LCD screens are comprised of a rectangular an^y of tiiousands of grid elements, referred to 
as pixels, each capable of displaying any one from a large number of color values from an RGB 
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color space, that when perceived as a whole, form an im^e. LCD screens are characterized by 
the number of horizontal and verticai pixels they contain. 

Each pixel in turn is composed of three individually addressable sub-components, refened 
to here as subpbcels. Most commonly, the three subplxels are rectangular red, green, and blue 
elements. In the nrrost common implementation, the three red, green, and blue subpixels are each 
assigned a luminous intensity value such that they blend together to give the entire pixel the 
appearance of the desired color. All of the pixels on an LCD screen blend together, in tum, to give 
the appearance of the desired image. 

The subpixels are considered individually addressable because the color value assigned to 
an individual pixel has a separate red, green, and blue color component, or luminosity value, which 
will be displayed, respectivety, by the red, green, and blue subpbcels of that pixel. Thus, the 
luminosity of each subpixel can be separately controlled by controlling the value of its associated 
color component luminosity value in the color value assigned to the pixel. 

in an LCD device and other "subpixel addressed" displays, such as color LED (including 
screens using organic light-emitting diodes (OLEDs)) or gas plasma displays, each individual 
subpixel has a fixed, Icnown position on the display. Many display devices, such as almost all 
cathode ray tube (CRT) displays are not subpixel addressable. For example, although each pixel 
of a CRT has an individual luminosity value for each of its red, green, and blue component colors, 
the exact physical location within each such pixel of the elements which generate the light 
associated vrith those different color values is nomially not known because it varies as a function of 
the individual phosphor pattern of the screen, the resolution of the horizontal and vertical scan, and 
the current exact state of the voltages which control the exact locations at which individual pixels 
are drawn on the screen. 

FIG. 9 illustrates a 12x12 portion of an LCD screen 900 that is comprised of a plurality of 
pixel rows (R1-R12) and pixel columns (C1-C12). Each intersection of a row and a column 
constitutes a pixel element Actual implementations of LCD screens can have an artDitrary number 
of rows and columns, though grids of 320 by 240, 640 by 480. 800 by 600, 1024 by 768, and 1280 
by 1024 are frequently seen. 
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Pixel R1-C1 Is contained wlttiin circle 910. Pixel R1-C1 is itself made up of tliree pixel sut)- 

components herein re1en«d to as subplxel elements. An expanded view of pixel C1-R1 Is shown 

as expanded pixel 920. Subplxel element 902 displays as red, subpbcel element 904 displays as 

green, and subpbcel element 906 displays as blue. The Individual subplxel elements 902, 904. and 

906 are approximately 1/3 of the width of a full pbcel and are equal In height to a full pixel. 

As illustrated in LCD screen 900, when anayed in a grid this causes the appearance of 
vertical color stripes down the LCD screen 900. This known arrangement of pbcels Is sometimes 
refenred to as vertical RGB striping. Other known arrangements lay the pucet elements out in the 
orthogonal direction such that horizontal striping results On which case rotating the screen by 90 
degrees will convert Is Into a vertically striped screen). 

In common usage, the luminous intensity of the three subpixel elements of a pkel are set 
such that the pixel is perceived by the human eye as being of the desired hue. saturation, and 
Intensity. The RGB subpb^l elements are used together to form a single colored pixel to represent 
a single sample of an image to be displayed. 

One aspect of the present Invention relates to the improvement of the readability of 
downloaded Web content, and other digital content Including text and images, on low-resolution 
screens, such as, for example, displays having column by row pbcel ratios of 320 by 240 or 240 by 
320 (in which case they can be rotated 90 degrees to have a resolution of 320 by 240). Many of 
the embodiments of the present invention discussed and shown in some detail map Image and text 
from a virtual layout resolutton of 640 by 480 pbcels onto a screen with a 320 by 240 pixel 
resolution. But the present invention can be used wtthotiier resolution screens. To give Just a few 
examples, it could be used to display content laid out roughly as it would look at a 1024x768 
resolution on a 512x384 resolution screen, or display content laid out roughly as it wouki look at a 
800x600 pixels on a 400x300 screen. In ottier embodiments, ttie invention can be used witii 
relatively low^resolution displays which have pbcel dimensions which are ottier tiian even fractions 
of ttie horizontal and/or vertical pbcel dimenstons common on personal computer screens. 

In general when we refer to a small resolution screen we mean a screen having a smaller 
resolution tiiat given digital content or a given layout of digital content woukl nonnally be Intended 
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to be displayed upon. By such smaller screens we also mean to Include portions of larger screens, 
such as windows on larger screens, that have such lower resolution. 

In FIG. 10, image content 105 and text content 107 represent a portion of the subplxel- 
optlnftod display 106 of FIG. 1. The image shown in FIG. 1 is a grayscale blowup of the actual 
color values associated with the subpixel-optimtzed display of both text and images. The portion of 
the Image content 105 contained within the rectangle 1000 is shown expanded at 1020 to make its 
Individual pixels easier to see. Correspondingly, a portion of the text content 107 contained within 
rectangle 1040 is shown expanded at 1060. 

It is important to note that the pMs shown at 1020 and 1060 represent whole pixels 
because the software used to generate the images 1020 and 1060 merely represents the 
grayscale oonesponding to the RGB color values associated with individual whole pixels. The 
subpixel blowups 1020A and 1060A are attempts to represent the intensity of each of the three 
subpixels associated vwth each pixel in the blowups 1020 and 1040, respecfively. 1020B is a 
blowup comesponding in scale and location to blowups 1020A and 1020. In it the pixel grid of the 
Image Is displayed in relatively bold lines, and the three subpixel divisions within each such pixel 
are shown in somewhat finer lines. This composite grid is superimposed on top of the original 
higher resolution color bitmap image 102 of FIG. 1 from which the pixelation patterns shown in the 
blowups 1020 and 1020A have been derived. In the particular images shown, the resolution of the 
color bitmap 102 is twice as high in both the vertical and horizontal direction as the whole pixel 
resolution in the image 105 shown at the bottom of FIG. 10. 

The blowup 1060B illustrates how a subpixel-optimized image of a font is produced by 
detemiinlng the portion of the area of each subpixel that is covered by a high-resolution font outline 
of the one or more characters involved. 

As can be seen by comparing the subpixel resolution blowups 1020A and 1060A to the 
conesponding whole pixel blowups 1020 and 1060, respectively, the display of subpixei-optimized 
representations of images and text at subpixel resolution provides better resolution. 

FIG. 11 provides a representation of the quality of the readability provided by an 
embodiment of the invention when displaying standard Web content on a 320 by 240 color display. 
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Bitmap 1100 at the top of the figure Is a grayscale, whole-pixel blowup of an actual bitmap 
produced from a standard 640 by 480 layout of a portion of a priceline.com web page. This image 
conresponds to the portion of the web page contained within the rectangle 1130 shown in the 
whole screen 320 by 240 image of the web page shown at the bottom of FIG. 11. Bitmap 1120 in 
the middle of the figure is a grayscale, whole-pixel blowup of the color bitmap of the same portion 
of of that 320 by 240 Image. 200B at the bottom of FIG. 11 represents a hand-held computing 
device that is functioning as a thin client browser of the type described with regard to FIG. 2. On 
the screen of this browser is shown a representation of a 320 by 240 subpbcel-optimized bitmap 
representing a 640 by 480 layout of the web page. Like the blowup 1020 of FIG. 10, the bitmap 
1130 shown at the bottom of FIG. 11 illustrates individual pixels with grayscale levels 
conesponding to the average luminosity of whole pixels. When this image is seen on a 320x240 
screen having vertical subplxel striping, as shown in FIG. 9, the actual Image appears to have an 
even higher resolution, as indicated by the blowup 1020A In FIG. 10. 

Any known algorithm for deriving subpixel-optimized images of color bitmaps can be used 
for the purposes of many aspects of the present invention. In one embodiment of the present 
Invention, the luminosity assigned to each given subpixel of a given color Is detemnined by the 
average intensity of that given color's value in each total or partial pixel of the source Image inside 
a rectangular window in the source image. This source image window conresponds to the area of 
a whole pixel in the scaled down image centered around the given subpixel. The average intensity 
assigned to the subpixel is calculated by multiplying the intensity of each source image pixel that 
totally or partially covers the source image window by the percent of that window's area covered by 
each such source image pixel. 

FIG. 12 illustrates the mapping of a reduced resolution dispia/s subpixel grid onto a 
portion of a higher resolution source image 102. This figure illustrates the subpixel grid 1210 being 
superimposed on a portion of the original higher resolution color bitmap 102 shown in FIG. 1. 
Circle 1220 encloses an area of that grid that con^sponds to one pixel in the intended lower 
resolution display device. The position and scale of the grid patterns is detemfiined by the 
relationship between Uie higher resolution source bitmap image and the pixel grid of the resulting 
subpixel-optimized images. The particular grid pattern 1210 shown in FIG. 12 represents a scaling 
from the pixel resolution of the color bitmap Image 1 02 to a display screen resolution that has one- 
half as many pixels in both the horizontal and vertical direction as the source image. An example 
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of this scaling is that of an image having pixelatlon appropriate for display on a 640 by 480 display 
be\ng scaled down for proportional display on a 320 by 240 display screen. Thus, each bold line 
division of grid pattern 1210 covers four pixels of color bitmap image 102. The dashed drcle 1220 
encloses one such bold line division that contains four higher resolution source pixels. 

FIG. 13 is an expansion of the nine bold line divisions (i.e., nine whole pbcels) centered on 
circle 1220 of FIG. 12. The pixel inside circle 1300 represents a single pixel of the intended display 
device. As FIG. 1 3 malces clear, each bold line division of grid pattern 1210 encloses four pbcels of 
the higher resolution source Image. The det^i of FIG. 13 also illustrates that each pbcel of the 
intended display device is made up of three color subpixeis, including a red, a green, and a blue 
subpixel, labeled "R", "G", and "B", respectively. 

FIGS. 14, 15, and 16, respectively, illustrate the positioning of the rectangular window area 
in the source image lipom which the luminosity of red, green, and blue colored subpixeis in the 
intended display device is detemnlned. The area of each such source image window conBsponds 
to the area of a whole pixel in the scaled down Image centered around the portion of the source 
image oonresponding to the subpixel whose luminosity it is being used to calculate. 

Rectangle 1400 of FIG. 14 encloses the area of the source image window used to 
calculate the luminosity of the red subpixel of the lower resolution display device. Similarly, FIGS. 
15 and 16 enclose tiie source image vinndows ttiat conrespond to ttie green and blue subpixeis of 
the Intended display de\rice, respectively. 

As stated above, the luminosity assigned to a subpixel of a given color is detemfiined by ttie 
following function, or an approximation tt^ercof. The luminoisity is set equal to the intensity of tiie 
supixers color in each pixel of ttie source image totally or partially within tiie subpixel's 
con'esponding source Image window, times ttie percent of ttiat window's area covered by each such 
source image pixel. 

In FIGS. 17, 18, and 19. tiie luminosity of tiie red. green, and blue, subpixeis Is a ftinctton 
of ttie respective color luminosities of whole or partial source Image pixels contained vJMn a 
source image window centered around tfie portion of ttie source image corresponding to a given 
subpbcel. This is shown in FIG. 17 for ttie red (R) subpixel. in which window area 1700 is centered 
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on the portion of the source image corresponding to that subplxel. Window area 1800 of FIG. 18 
Illustrates the same for the green (6) subplxel, and window area 1900 of FIG. 19 Illustrates the 
same for the blue (B) subpbcel. 

As a result of the shift between the source Image windows for each subpbcel, the color 
value derived for each subpbcel represents the subpbceFs con'esponding color in a portion of the 
source image corresponding to the location of each subpbcel, itself, rather corresponding to the 
location of Ite pbcel as a whole. As a result, this use of difierent source image windows for different 
subplxels of a given pixel Increases the spatial resolution of the resulting image. 

In the embodiment of the Invention shown in FIGS. 17, 18, and 19, the detennination of 
which pbcels fell within a subpbcel's source image window and the percent of that v\dndow each 
such pixel covers is made by an approximation based on the percentage of horizontal and vertical 
scan lines covered by such source image pixels. In FIG. 17, the color value of the red subplxel Is 
determined as a result of the percentage of a horizontal coverage line 1720 and a vertical 
coverage line 1740 covered by Individual source image pbcels. times the red color value of each 
such pbcel. The same Is true, for respective cotor values, for the scaled image's green (6) subplxel 
of FIG, 18 and its horizontal and vertical coverage lines 1820 and 1840 respectively, and the 
scaled Image's blue (B) subpbcel of FIG. 19 and its horizontal and vertical coverage lines 1920 and 
1940, respectively. It should be noted that horizontal coverage lines 1720, 1820, and 1920 are 
Intended to represent vertical positions just above or below the vertical midpoint of their 
con'esponding rectangular area. This Is so the coverage line will not exactly equal that location in 
the source image that represents the border between vertical pbcels. In the same manner, the 
vertical coverage line 1740 is intended to represent its horizontal position just to the left or the right 
of the horizontal midpoint of the rectangular area 1700. 

The above defined coverage lines represent an embodiment of an aspect of the Invention 
that relates to the use of a continuous function, or a reasonably high resolution (such as 5 or more 
bit resolution) equivalent thereof, to detemiine the extent to which the area of an original Image 
associated with a given color subpbcel is covered by a given color or shape. In continuous 
coverage functions, this coverage is detennined, not by sampling, but rather by a mathematical 
function that detemilnes boundary locat'ons at which the given coverage starts and stops in one or 
more dimensions, and calculates coverage as a function of lengths or areas between one or more 



wo 02/088979 



PCT/US02/14380 



-28- 

such boundaries or between such boundaries and the boundary of the source image window 
associated v\nth a given subplxel. 

In the embodiment of tiie invention shown in FIGS. 17. IS, and 19 and in FIGS. 30, 31, 
and 32 the calculation of this continuous coverage tiincHon is sped up by estimating the area of 
each source image pixel that is in a given subpbcel's conresponding source image window area by 
detennining the portion of one or more scan lines within the rectangular area that is covered by 
each of one or more of the source image's pbcels vrithin the window. The percent of the total length 
of the window's scanning lines that is covered by a given pixel is multiplied by the value of the 
subpixel's color in that pixel. Such producte are summed over all pixels that cover any of the 
window's scan lines to produce the subpbceFs color value. This is how a "line coverage" type of 
continuous coverage function can be used to detemnine the luminosity of a subpbcel when creating 
scaled images of color bitmaps. 

FIGS. 20, 21, and 22 illustrale the use of single horizontal and single vertical coverage 
lines within the source image \Mndow 2000 associated with the red (R) subplxel in the lower 
resolution display screen. In FIG. 21, the coverage value assodated with horizontal scan line 2020 
is the summation of: 

-the red value of the pixel covered by bracket 2120. times the portion (1^) of horizontal 
scan line 2020 covered by braclcet 2120, plus 

-the red value of the pixel covered by braclcet 2140, times the portion (1/2) of horizontal 
scan line 2020 covered by bracket 2140, plus 

-the red value of the pixel covered by bracket 2160, times the portion (1/6) of horizontal 
scan line 2020 covered by bracket 2160. 

In similar fashion the coverage value associated with vertical scan line 2040 is the 
summation of: 

-the red value of the pixel covered by bracket 2220. times the portion (1/2) of vertical scan 
line 2040 covered by bracket 2220. plus 
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-the red value of the pixel covered by bracket 2240, times the portion (1/2) of vertical scan 
line 2040 covered by bracket 2240. 

The total coverage value for the red subpixel is one half of the coverage value cateulated 
for the horizontal scan line plus one half of the coverage value calculated for the vertical scan line. 

Similarly, FIGS. 23, 24. and 25 Illustrate the use of single horizontal and single vertical 
coverage lines vdthin the source image window 2300 associated with the green (G) subpixel In the 
lower resolution display screen, and FIGS. 26, 27, and 28 illustrate the use of single horizontal and 
single vertical coverage lines within the source image vwndow 2600 associated with the blue (B) 
subpbcel in the lower resolution display screen. 

FIG. 29 Is highly simplified pseudo-code representation of an algorithm 2900 for deriving 
scaled subpixel-optimized images from a source bitmap image using line coverage of the type 
described above with regard to FIGS. 17 through 28. 

This algorithm performs a loop 2901 for each pixel row of the output, Image (l.e, the 
scaled, subpbcel-optimlzed Image). 

This loop perfonns an inner k3op 2902 for each pbcel In its cun'ent row. For each such pbcel 
the loop 2902 perfbnns a loop 2904 and a fundion 2914. 

The loop 2904 is comprised of an interior loop 2906 that is performed for each of the 
subpbcers scan lines, such as the scan lines shown in FIGS. 17 through 28. 

The loop 2906 Includes a function 2908 and a loop 2910. The function 2908 calculate 
each Intersection between that scan line and a pixel boundary. Nonnaily. such intersection 
cateulations and the other calculations In this algoriUim are perfonned with limited accuracy, such 
as for example 6 to 8 bite of accuracy, to reduce the storage and computational requiremente of 
such computation. 
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Then a loop 2910 perfbnns a function 2912 for each portion of a scan line that occurs 
between two scan line ends, a scan line end and a pixel boundary, or two pbcel boundaries. 
Function 2912 adds to a coverage value associated with the current subpixel of the loop 2904 a 
multiple of the percent of that scan line covered by cun^ent portion of loop 2910 times the 
component color value of the pbcel covering that portion conesponding to the color of the current 
subpixei, all divided by the number of the subpbcel's scan lines. 

Once the loop 2904 has calculated the subpbcel luminosity value for each subpbcel of the 
cun^ent pixel, function 2914 sets the cunent pbcel's color value equal to a color having a compound 
RGB value with red, green, and blue values equal to those calculated subpbcel luminosity values. 

In different embodiments of the invention different length color values can be used, such 
as 24 bit, 16 bit, or 12 bit color values. Although the system can be used with a limited color 
palette, it worlcs best with tme-color colors, which have at least 4 bits of variability for each of the 
red, green, and blue subpixels. 16 bit color, which commonly allocates 5 bits for red and blue and 
6 bits for green (because of the eyes' greater sensitivity to green), provides even better visual 
results. 

Although the embodiment of the invention described above with regard to FIGS. 17 
through 28 mal<es use of a single horizontal and a single vertical coverage scan line, other 
embodiments of this aspect of the invention may have more scan lines and/or have scan lines in 
orientations other than horizontal and vertical. 

FIG. 30 illustrates the use of two horizontal coverage lines and two vertical coverage lines 
within a source image window 3020 that can be used to estimate the color coverage associated 
with a red (R) subpixel of a subpbcel-optimized image. 

FIG. 31 illustrates the use of two diagonal coverage lines within a source image window 
3120 associated vrith a green (6) sut)pixel of a subpbcel-optimized image. 

FIG. 32 illustrates the use of two diagonal coverage lines, a horizontal coverage line, and a 
vertical coverage Dne within a source image window 3220 associated with a blue (B) subpixel of a 
subpbcei-optimized image. 
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Of course each of the arrangements of coverage lines shown in each of FIGS. 30 through 
31 can be used on either red, green, or blue pixels. 

FIGS. 33 through 38 illustrate that the line coverage method of calculating subpixel 
luminosity values can be applied to a broad range of different scalings between the size of a 
source image and the resulting subpixekptimized image. This is true because the line coverage 
method measures line coverage at a feiily high resolution, compared, for example, to many 
sampling techniques. This means that it does a relatively good job of measuring the coverage of 
pixels which are only partially in a subpixeFs source image window, as will often result when using 
scaling factors that are non-integer ratios. 

In one embodiment of this aspect of the Invention a seven bit resolution is used in 
calculating line coverage, which produces satisfactory results. iHigher or lower resolutions can be 
used, but it is preferred that the line coverage resolutions be higher than the two to four bit per 
dimension resolution commonly used in techniques that measure coverage by sampling coverage 
within a subpbcel's source image window at an anay of sixteen (4x4) to two-fifty-sb( (16x16) points. 

FIG. 33 illustrates the coverage of two horizontal coverage lines by various source image 
pixels within a source image window associated with the blue (B) subpixel for a mapping from a 
source image resolution to a destination pixel-optimized image having half as many horizontel 
pixels and vertical pbcels. FIG. 34 does the same for the two vertical coverage lines used with such 
subpbcel luminosity calculation scheime. Thus, FIGS. 33 and 34 illustrate an Integral ratio between 
the number of pbcels In the source and the reduced images. 

FIGS. 35 and 36 illustrate the coverage of horizontel and vertical scan line, respectively, by 
pixels of the same source image for a scaling tector In which the reduced subpbcel-optlmized image 
has only about 40% as many horizontel and vertical pbcels as the source image. 

FIGS. 37 and 38 illustrate ttie same for a scaling factor in which the subp]xel<>ptimized 
image has about 66.66% as many horizontal and vertical pbcels as ttie source image. 
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It can be seen that the scan line coverage technique shown In FIGS. 33 through 38 
provide an accurate estimate of the percent of each source image window covered by each source 
Image at each of different scaling, with relatively little computation. 

FIGS. 39 and 40 illustrate the geometries associated an 'area' type of continuous 
coverage function. In some embodiments of the invention, the percent of a given subpixers source 
image window covered by each of its associated source Image pixels Is calculated, not by the line 
coverage approximation described above, but rather by an actual calculation of the area of that 
part of each such source image pixel that lies within the subpbcei's source Image window. For 
each such source pixel, the component color value of the pixel corresponding to the color of the 
cunBnt subpixel is detemiined. The luminosity value for each subpbcel is then calculated by 
summing the multiples of the percentage coverage value and the value of the subpixefs color for 
eadi source Image pbcel that appears in its source image window. 

FIG. 39 illustrates the source image window area 3900 associated with a blue (6) subpixel. 
Source pixel 3920 is contained within source Image window 3900, as are portions of eight other 
source pixels. The percent of the source image window 3900 covered by a source pixel 3920 is 
calculated by taking the ratio of the area of the hatched portion 4020 of FIG. 40 over the area of 
the whole source image window 4000. Similarly, the percent of the source image window 4000 
covered by the other source pixels contained within It are calculated by taking the ratios of their 
area within the source image window, as indicated by differently hatched areas of the window 
4000, over the total area of that source image window. 

FIG. 41 provides a highly simplified pseudocode representation of an algorithm 4100 that 
can be used to implement an area coverage function of the type which Is been discussed with 
regard to FIGS. 39 and 40. 

The algorithm comprises a loop 4102 which is perfonned for each pixel now in the sub 
pixel-optimized image to be produced. For each such row the loop 4102 performs an inner loop 
4104 for each pixel In that row. 

This Inner loop 4104 Is comprised of a loop 4106 and a function 41 16. The loop 4106 is 
perfomfied for each subpixel in the cunrent pixel of the k)op 4104. This inner loop 4106 Is 
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comprised of a function 4108 and a bop 41 10. The function 4108 detemilnes which pixels of the 
source image are in the source image window associated with the subptxeli as described above. 
Once this is done the loop 4110 is performed for each such source image pbceL 

The loop 4110 is comprised of a function 4112 and a function 4114. The function 4112 
calculates the percentage of the subplxei's source image window area covered by the current 
source image pixel of the loop 4110. Then step 4114 adds to the luminosity value being calculated 
for the cunent subpbcel of the loop 4106, the multiple of the percentage of the subpixei's source 
image window area covered by the cunent source image pixel, times the source image pbcei's color 
component value corresponding to the color of the current subpbcel. 

Once the loop 4106 has been perfonned for each subpbcel in the current pixel, function 
41 16 sets the cunient pixel's color value equal to a color having RGB color component values 
con-esponding to the red, green, and blue subpM luminosity values calculated by the loop 4106. 

FIGS. 42 through 53 relate to aspects of the invention relating to bicolor subpixel- 
optimlzed images. 

A "bicolor* image is one in which individual pixel colors range between two different color 
values. Commonly these two different color values will be black and white, and the pixels of the 
sourte and subpbcel image will have values limited to black, white, or a grayscale value in between. 
In some embodiments, however, the two different color values can represent any uniform 
foreground and background colors, and colors intemiediary between them. Bicolor images are 
often used to represent text, because the display of text is often bicolored, involving a foreground 
color and a background color. But bicolored images can also also be used to represent other 
bicolored shapes, bicolored bitmaps, portions of multicolored bitmaps which are bicolored, or 
multicolored bitmaps which are to be represented with bicolored output, such as a grayscale 
image. For example, a multicolor source image can be treated as a corresponding grayscale 
image, merely by treating each of its pixels as having a grayscale value corresponding to the 
average luminosity of each of Its three color components. 
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The advantage of using such bicolored subpixel-optimized output images is tliat they often 
can provide a higher spatial resolution than multicolor subpixekptimized output images. Such 
higher resolution is allowed v\^ere the bicolors are black and white, greyscale values, or opacity 
and transparency, because each subpixel can represent both the foreground and baclcground of 
such blcolor pairs equally as well as any othersince each color of each such bicolor has equal 
components of red, green, and blue. Except for the need to perform color balancing, as is 
described below, each subpbcel's luminosity can be detemfiined as a function of the extend to 
which the portion of the source image oonesponding to its own area in the output image is covered 
by a foreground or a background color. This use of a smaller source window, l.e, one 
conesponding to a subpbcel's size rather than to a pixel's size, allows a more accurate spatial 
representation of the source image. 

Where the foreground and background colors are not black and white, the resoiutton 
produced by bicolor subpbcel-optimized images will be best if the foreground and background color 
eadi have red, green, and blue values that are relatively equ^ in luminosity, but wHh the average 
luminosity of the foreground and background color as different as possible. In fact, In some 
embodiments of aspects of the invention relating to bicolored subpixel optimized images one or 
both of the output bicolors are changed from the con-esponding input bicotors by being shifted 
toward a corresponding grayscale color to improve the spatial resolution of the output image. 

The extent to which a subpixel of a bicolor subpixel-optimized output image is to 
display the foreground color is sometimes represented by an alpha, or opacity, value. Such an 
alpha values indicates the extent to which the subplxeFs luminosity should correspond to the its 
color component in the foreground color or in toe background color. An alpha value of one means 
the subpixel's color component value should equal the coniesponding color component in the 
forground color. An alpha value of zero means it should equal the corresponding color component 
in the background color. An intermediary alpha values means the subpixel's color component 
value should be a weighted blend of the conesponding color components in both the foreground 
and backgound colors. Once a subpbcel-optimized bitmap is represented in terms of alpha values 
it can be used to represent bicolor images of a given pattem using different forground and 
background colors. This is commonly used to represent font shapes, since in the presentation of 
fonts the bitmap pattem of a given character^ont shape at a given size is often displayed vrith 
different foreground and background colors. 
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In some embodiments of aspects of the invention relating to bicotor subplxel optimizations 
of bitmap images a scaled subpixel-optlmized image of a bitmap Image Is produced by associating 
a foreground or baclcground blcolbr coverage value with each subplxel of the scaled image as a 
function of: (a) the ratio of the foreground or background color for each source image pixel In a 
source image window conesponding to the area of the subplxel; (b) the percent of that vrfndow 
covered by each such source image pixels; and (c) a color balancing function that distributes 
subpbcet coverage values to reduce color imbalance. In cases In which a blcoior output image is 
being produced for either a grayscale or a multicolor input image, the coverage values calculated 
for individual subpixeFs can be derived as a function of the v*oIe pbcel luminosity of source Image 
pixels which cover its source image window. In some embodiments, the extent to which a given 
luminosity value associated with a given subpixel's source image window is distributed to other 
subpixels is a function of extent to which the lunfdnpsity value causes a color Imbalance. 

FIGS, 42 through 44 iflustrate a method of determining the luminosity of each subpbcel of a 
grayscale bicolored Image. In FIG. 42. rectangle 4200 encloses a window of the source Image that 
is associated with the red (R) subpixel of the scaled image. The luminosity to be associated with 
such red (R) subpixel is a function of the whole pixel luminosity of the one or more source image 
pixels that cover the source Image window 4200, multiplied, respectively, by the percent of the 
source image window covered by each such source image pixel. Any known method for calculating 
or estimating such coverage percentages can be used. 

In the embodiment illustrated in FIG. 42 source image window 4200 has associated with it 
two horizontal scan lines 4210 and 4220 and two vertical scan lines. FIGS, 43 and 44 Illustrate the 
coverage lines for the source image windows 4300 and 4400 for green and blue subpixels, 
respectively. As before, to estimate the extent to which the source image vwndow areas are 
covered by a source pixel, a mathematical function tiiat detemilnes boundary locations at which 
the given coverage starts and stops along each scan line is mn. Coverage is calculated as a 
function of tiie lengttis between one or more such boundaries or between such boundaries and Uie 
boundary of tiie source image window associated witii the given subpixel. This can be done In a 
manner similar to tfiat described above in FIG. 29. 
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When calculating bicolor subplxel-optimlzed images, color imbalances may occur. This is 
because the bicolor methods is attempting to produce an output image in which each whole pbcel 
has a color value in the spectrum between the two bicolors (usually black or white), but the 
coverage values of a pixel's Individual red, green, and blue subpixels is detemoined by the percent 
of foreground color in each such subpixel, meaning that the color of individual ou^ut pixels would 
often have no relation to the desired bicolor spectrum (usually grayscale), In the absence of such 
color balancing. 

For example, In a grayscale image, if the source image makes a transition from totally 
white to totally black at a tocation corresponding to the boundary between a red and green 
subpbcel in a subpbcel optimized output image, the conesponding pixel In the output image will 
have a red subpixel coverage value that would tend to cause that subpixel to be turned totally on, 
and green and blue subpbcels coverage values that would tend to cause those subpixels to be 
turned totally off. This would result In a visible red color for the pixel, even though, in this example, 
the output image is supposed to be a grayscale Image. 

FIGS. 45 threugh 47 illustrate how a traditional linear color balancing method of a type 
used in the prior art to color balance subpixel coverage values calculated from the rasterization of 
font outlines can be used to color balance coverage values produced from bicolored bitmaps. 

FIG. 45 illustrates a set of grayscate source image pixels under an RGB grid 4600. Grid 
4600 has tour pixel areas enclosed in bold line divisions. Each such pixel area Is associated with a 
whole pixel in a subpixel addressable screen on which the output Image is to be displayed. Each 
pbcel area is further divided Into three areas associated with the subpixels of associated pixel on 
the subpixel addressable screen. Subpixel-associated area 4610 Is associated with the red (R) 
subpbcel, subpixel-associated area 4612 is associated with the green (G) subpbcel, and subpixel- 
associated area 4614 is associated with the blue (B) subpbcel. Subpixel-assoc^ted areas 4616 
through 4632 are associated with their respective display screen pixels. 

Subpbcel-associated areas 4614 through 4630 are covered in whole or in part by source 
image pbcels having nonwhite luminosity values corresponding to various degrees of the 
foreground color, which in this case is black. The total nonwhite luminosity value of the source 
image pbcels In eac^ of the subpbcei-assodated areas 4614 through 4630 is mapped into 
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corresponding sub-pixel areas in the RGB grid 4700 of FIG. 46. The height of the hatched afea 
within each of the subpixel areas 4744 through 4760 is detennlned by the total nonwhHe luminosity 
values of the conespondlng sub-pbcel areas 4614 through 4630. 

The bottom half of FIG. 46 illustrates the use of a center-weighted, symmetrical color filter, 
which can be used to distribute the coverage value associated with the subpixel 4750 over five 
subpixels centered around the subpbcei 4750. Three ninths (3/9^) of the coverage value of 
subpixel 4750 is distributed into sub-pbcel 4750, Itself. Two ninths (2/9^) of the coverage value of 
the subpixel .4750 is distributed into the subpixels 4748 and 4752 which are Immediately to its left 
and to its right, respectively. To complete the color distribution of subpbcei 4750, one ninth (1/9^) 
of its coverage value is distributed into subpixels 4746 and 4754, which are two sub pixels to the 
left and two sub pixels to the r^ht, respectively of the subpixel 4750. 

In general, color balancing distributed color values within a neighborhood of nearby pixels 
in which the neari)y pbcels are nomially within a distance of no more than one pbcel from the 
subplxel's whose color is being distributed, although in some embodiments that distance might be 
as large as two pbcels. 

FIG. 47 illustrates the result of the symmetrical center-weighted color balancing filter of 
FIG. 46 when it is applied lineariy to the coverage value calculated for each of the subpixels 4740 
through 4762 shown in the top half of FIG. 46. 

In FIG. 47 the coverage value associated with each subpixel 4744 through 4760, shown at 
the top of FIG 47, is distributed using a color balance filter which distributes Ite coverage value In 
ttie same proportion to ite own subpbcei and to the two subpbcels to the left and right as Is shown in 
FIG. 46. The central grid 4802 of FIG. 47 graphically Illustrates the size of the contribution that 
such a distribution malces to each of the subpixels 4740 through 4762. The distribution associated 
with each of the given subpbcels 4744 through 4760 Is centered in a vertical column located directly 
below its respective subpixel. 

RGB subpixel grid pattern 4804 shown at the bottom of FIG. 47 Illustrates the lunrtfnosity 
value that is calculated for each subpbcei 4740 through 4762 by summing all the contributions which 
have been made to it by all of the coverage value distributions illustrated in the center panel 4802. 
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To complete the methodi the luminosity values of the red, green, and blue subpixels of each pixel in 
grid 4804 are used as the three component color values that specified the color of each such pixel. 

While this linear method does reduce the color imbalance of ttie scaled Innage, it does so at 
the expense a substantial reduction in spatid resolution. This can be understood by comparing the 
values in RGB grid pattern 4804 at the bottom of FIG. 47, which represents the subpixel luminosity 
values in the subpbcel-optimized output image, to the values in RGB grid pattern 4800 at the top of 
FIG. 47, which represents the luminosity, or foreground color coverage, of the source image pixels 
coaesponding to the subpixels of that output image. As can be seen by FIG. 47, the spatial 
resolution of the output image is smeared relative to the spatial resolution of the source image. 

The present Invention includes an innovation that provides similar color balancing of 
subpixel optinnized output images, but often with much less smearing of the output image. It does so 
by using a non-linear color balancing filtering method. A mettiod of applying this non-linear filtering 
tsiUustratedin FIGS. 48 and 49. 

FIG. 48's RGB grid pattern 4900 is a duplication of the RGB grid pattem 4700 of FIG. 46. 
Once again, the total luminosity, or foreground color coverage, values of the source image pixels 
that correspond to an associated subpbcel are represented by hatched areas. 

The first phase in this non-linear color balancing method is illustrated in FIG. 48. As before, 
RGB grid pattem 4900 is divided into fourths by the bold line divisions and each fourth, bracketed 
portions 4902, 4904, 4906, and 4908, is associated with a whole pixel of the scaled, or output, 
image. Each of tiie pixel areas 4902, 4904, 4906, and 4908, is further dwided into subpixel areas 
coResponding to subpixels In the output image to be produced. For each pbcel, a determination is 
made of which of its sub pixels has the lowest subpixel luminosity, or foreground coverage value 
and a luminosity (or alpha) value equal to this minimum luminosity/coverage value is added to a 
luminosity/alpha value which is being calculated for each subpixel of the pbcels 4912, 4914, 4916, 
and 4918 of RGB grid pattem 4910, shown in the bottom half of FIG. 48. 

In the top half of FIG. 48 the hatched line 4920 indicates the minimum luminosity/coverage 
value of the pixel area 4902 Is zero, since the first two subpixel-associated areas have luminosity 
values of zero. Thus, the step shown in FIG. 48 sets the luminosity/alpha value for the red, green. 
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and blue subpixel areas of pixel 4912 to zero. In like fashioHi the minimum luminosity value of pixel 
area 4904 Is detemilned by the value of the ted subpixel area 4922 of the pixel 4904. This 
minimum luminosity value is mapped into the conBsponding pixel area 4914 in the bottom half of 
the figure. Similarly, the minimum luminosity values of pixel areas 4906 and 4908 are mapped into 
pixel areas 4916 and 4918 in the bottom half of FIG. 48. The resulting partially calculated 
luminosity/alpha values after the completion of this step are represented by RGB grid pattem 4910 
at the bottom of FIG. 48. 

The second phase of the non-linear color balancing method is illustrated in FIG. 50. In this 
example of the second phase, the portion of the luminosity/coverage value of each subpixel that is 
in excess of the pixel's minimum luminosity/coverage value is mapped into the RGB grid pattem 
4910 by utilizing a color balance distribution filter of the type shown above with regard FIGS. 46 and 
47. 

The pbcel grid 5000 at the top of FIG. 49 conresponds to the pbcel grid 4900 at the lop of 
FIG. 48 (and has the same sub pbcels 4740 through 4762) except that it represents the portion of 
each subpixePs luminosity/coverage values(shown in high frequency hatching)FIG. that remains 
after the value of the minimum subpixel luminosity/coverage value for the corresponding pixel 
(shown in low-frequency hatching) has been subtracted from it. FIG. 

The subpixel grid 5002 in the middle of FIG. 49 corresponds to the similarty shaped pbcel 
grid 4802 in the middle of FIG. 47, except that in It only the excess portion subpixel 
luminosity/coverage values shown in the top of FIG. 49 are distributed using color balance filters of 
the type shown In FIG. 46. As can be seen in this portion of the FI6.figure, the excess 
luminosity/coverage value for each subpixel Is distributed to its own subpbcel, to two pixels to the 
left, and to two ptols to the right using the same proportional filter shown in FIG. 46. 

The portion of FIG. 49 near its bottom labeled by the numeral 5004 shows the total of such 
excess luminosity/coverage value which is distributed to each subpixel 4740 through 4762 In this 
example of the non-linear method. The total excess luminosity/coverage value calculated for each 
subpixel is added to FIG.the minimum luminosity/coverage value that been previously added to that 
subpixel by the step illustrated in FIG. 48, as illustrated at the bottom of FIG. 49 to produce the totel 
luminosity/alpha value to be used for each subpixel in the output image. 
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To complete the non-iinear color balancing process, the luminosity/alpha values summed 
for each subpixel of RGB grid pattern 4910 are used to detemiine the conespondlng red, green, 
and blue, color component values of its associated pixel. The red, green, and blue color values of 
each Individual pixel in such a display may not be equal, but the total of the red. green, and blue 
color values in any neighborhood of five or so adjacent subpixels of a pixel row should be 
substantially equal, or balanced. 

A comparison of the results achieved by use of the linear method and the non-iinear color 
balance filtering method is illustrated by FIGS. 50, 51. and 53. 

FIG. 50 illustrates the original unfiltered source pixel luminosity/coverage values as first 
mapped into RGB grid patterns 4700 of FIG. 46 and 4900 or FIG. 48. 

FIG. 51 lllusbBtes the result of the non-linear filtering method, as shown at the bottom of 
FIG.49FIG.FIG. 

FIG. 52 illustrates the result of the linear filtering method, as shown in RGB grid pattem 
4804 of FIG. 47. 

As FIG. 51 shows, the output of the non-linear color balancing method more closely 
resembles the original spatial distribution of luminosity/coverage values of FIG. 50 than does the 
result of the linear method, as shown in FIG. 52. The luminosity values produced by the non-linear 
method of FIG. 51 are significantly less blunred, and, thus, provide a higher visible spatial resolution 
than the output produced by the linear method. This is because the non-linear method seeks to 
perfomfi color balance distribution, which has the detrimental effect of blumng spatial resolution, only 
on those portions of subpixel luminosity/coverage values which need such distribution in order to 
prevent color imbalance. This can be seen by comparing ttie total of subpixel luminosity/coverage 
values distributed to each subpixel using ttie non-linear metiiodFIG., as indicated by the numeral 
5004 of FIG. 49, with the conespondlng total which is distributed to each subpbcel using the linear 
method, as indicated by the entire crosshatched area shown for each subpixel at the bottom of FIG. 
47. 
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FIG. 53 provides a highly simplified pseudo code description of one implementation of a 
method of producing a subpixel-optimlzed bicolor output bitmap using the non-linear color balancing 
method. 

The algorithm 5300 m this figure is comprised of a loop 5301 which is perfomned for each 
pixel row in the image. This loop perfomis two subloops 5302 and 5322 for each pixel row. 

The loop 5302 perfomns a loop 5304, a function 5314 and a loop 5316. 

The loop 5304 is performed for each subpixel in the current pbcel of loop 5302. For each 
such subpixel it perlbnns a function 5306 and a loop 5308. 

The function 5306 detemnines which pbcels of the source Image are in a window portion of 
the source image corresponding to the subpixePs area in the scaled image. This can be perfonn^d 
by any known coverage calculation or estimation function, including the ones described above with 
regard to FIG. 17 through 44. 

The loop 5308, comprised of functions 5310 and 5312, is performed for each source im^e 
pixel that is totally or partially contained within the current subpixeFs source Image window. 
Function 5310 calculates the percent of the source image Window's area covered by the source 
image pixel's area. Function 5312 adds to a luminosity/coverage value calculated for the cument 
subpb(el, the multiple of the percentage of the window area covered by the source image pbcel time 
the source image pixel's average foreground color Intensity. 

In cases in which the bicolor image is a grayscale image, the foreground color intensity can 
conBspond to either the luminosity, or the Inverse of the luminosity, of each whole source image 
pixel. If the source image is a multi-color image, the average luminosity value of each source image 
pixel's color components can be used to detemiine a luminosity value that can be used for the 
source image pixel in calculating its foreground color intensity for the purposes of function 5312. 

Loop of 5304 can be used to detemiine the luminosity/coverage values of the type shown at 
the top of FIGS. 46. 
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Once the loop 5304 is been perlbmied for each subpixel in the cument pixel, function 5314 
finds the minimum subpixel luminosity/coverBge value which has been so calculated for the cunent 
pixel, as is illustrated in the top half of FIG. 48. 

Once this has been done the loop 5316. comprised of functions 5318 and 5320. is 
perfonned for each subpixel in the cunent pixel. 

Function 5318 sets a luminosity/alpha value being calculated for the subpixel to the 
minimum subpixel iuminosity/ooverage value detennlned for the pbcel by the funcfion 5314, 
somewhat as Is indicated In the bottom half of FIG. 48. 

Function 5320 distributes the portion of the subpixel luminosity coverage value that exceeds 
the pbcers minimum subpixel luminosity/coverage value to the luminosity/alpha values being 
calculated for the subpi)oel and adjacent subpixei's in the cunent pixel row using a color balance 
distribution fitter, as Is indicated in FIG. 49. 

in one embodiment of the invention, If the total of such distributions made to a given 
subpb^l exceeds the maximum allowed luminosity/output value, the subpixei's lumlnosity/aipha 
value is limited to that maximum value. Although this clipping causes some color imbalance, the 
Inventors have found the resulting Imbalances to be hardly noticeable. 

Once lumlno^/alpha values have been calculated for each subpbcel in the row and the 
loop 5302 has been completed, the loop 5322 causes a function 5324 to be perfonned for each 
pixel In the row. This function sets the pixel color value equal to a color having a compound RGB 
value with red, green, and blue component values corresponding to the luminosity/alpha values 
calculated for the pbceFs red, green, and blue sub pixels, respectively. 

FIG. 54 illustrates an aspect of the present invention in which the user of a display device 
can dynamically make tradeoffs between the extent to whidi a subpixel-optimlzed images produced 
from a source images is produced by a multi-color subpixel optimization process or a bicolor 
subpixel optimization process. An output image produced by the multi-color subpixel optimization 
can represent scaled down color images with reasonable color accuracy, whereas the bicolor 
subpixel optimization in this example can produce only grayscale output Images. But in some 
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cases, such grayscale output images will have a more accurate spatial resolution and. particularly 
where the source image has black and whtte portions with sharp edges, less perceptible color 
imbalance than an output image produced by the multi-color subpixel optimization. 

A Color bitmap image 5400 may be scaled and subpixel optimced by utilization of functions 
5410 and 5430, which use a bicolor subplxekptindzation method, such as one of those described 
above with regard to FIGS. 42 through 53, to produce a scaled and subpbcel-optimized grayscale 
bitmap 5440. Color bitmap image 5400 may also be scaled and subpbcei optimized by a process 
5420 tfiat uses a multicolor subpixel-optimization method, such as one of those described above 
with regard to FIGS. 17 through 40, to produce a scaled and subpbcel-optimized color bitmap 5450. 

According to an embodiment of the present invention shown in FIG. 54, the user of the 
display device can manipulate a control apparatus of the display device, such as a pointing device, 
keyboard, or other input device, according to process step 5460 in order to achieve a blend of the 
color bitmap 5450 and the grayscale bitmap 5440. The can be done, for example, by manipulation 
of a slide bar. Process step 5480 receives the grayscate bitmap 5440 and the color bitmap 5450 
and the user selected oolor/grayscale tradeoff infonmation and btends the color values of the 
conesponding pixels fifom the grayscale and color bitmaps 5440 and 5450, weighing color values 
from each as a functton of the user selected cok3r/grayscale tradeoff 5460. 

In some embodiments of the type shown in FIG. 54, if the user selecb a tradeoff value at - 
either extreme of the ootor/grayscale spectrum, the process can reduce computetton by only 
calculating the bitmap 5440 or 5450 which comesponds to that selected extreme. 

A benefit of this aspect of the present Invention is that the user of a display device can favor 
color balance and/or positional accuracy when that Is most Important or color accuracy when that is 
most important or simply vary the tradeoff selection to find a more easily readable display. 

Not all aspects of the present invention require subplxet-optimized text, and many of those 
that do can use prior art methods of creating subplxel-optimized bitmap's of font shapes. However, 
some aspects of the invention relate to Innovations in methods of making font bitmaps. 
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FIGS. 55 through 97 relate to aspects of the invention relating to the fomiing and using 
subpixel-optlndzed font bitmaps 

FIG. 55 illustrates a font outline 5500, in this case an outline of a capital letter "B" in a 
Times Roman font The outline is shown superimposed over a subpixel grid 5502. which is 
composed of a plurality of individual whole pixels 5504, each of which Includes a red, blue, and 
green subpixel, 5506, 5508, and 5510, respediveiy. 

The font outline shown In FIG. 55 is one that could be used for display at relath^ely nornial 
text sizes, indicating that the invention's method of subpixel optimizing character-font shapes is 
applicabte across a broad range of applicaflons and is not Bn^ed to small screen displays of the 
type shown at the bottom of FIG. 11. However, when this aspect of the invention is applied to 
small screen displays and/or the display of font at very small pixel sizes, it is prefenred that the 
fonts used be optimized for display at small sizes, such as In some embodiments ten pixels per em 
or less, or eight pixels per em or less. 

FIG. 56 illustrates a font which has been optimized for such small display on subpixel 
addressable screens. FIG. 57 shows the same bitmap at twice the size. Unfortunately the 
printouts of the bitmap shown in FIGS. 56 and 57 display the average luminosities of whole pixels 
and fall to capture the higher resolution made possible when such a bitmap is shown on a subpixel 
addressable display. 

The font shown In FIGS. 56 and 57 have been produced by a hinting process which shifts 
selected boundaries of individual font outlines to pixel boundaries, subpixel boundaries, and 
horizontal and vertical dimension's intemnediary between subpixel boundaries. Such high 
resolution hinting is used in order to achieve optimum readability on subpbcel displays. It Is done 
by having a font designer view subpixekptimized bitmaps of individual characters with various 
hinting values until he or she feels relatively satisfied that the character is as clear as possible 
when display at such a small font size. As those knowledgeable of font hinting vwll understand, a 
font can have hints which dictate the alignment of Individual portions of a font outline across all size 
renderings of that font, and special hints which are applied for the character-font shape at certain 
pfacel sizes. The font shown in FIGS. 56 and 57 have been hinted to optimize their display at eight 
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pixels per em, and some of them have specific hints which are to be applied only at such small 

sizes. 

In fact, most of the fonts in the 320 by 240 pixel resolution screen shots shown in the 
figuiBS of this application are of 8 pixel per em fonts that have been specificaliy hinted for display at 
that size. These fonts allow a relatively large amount of web text to fit on a small screen, while 
allowing a relatively high level of readability. These fonts allow the large majority of lower case 
characters to be represented In four pixel columns or less, Including space that separates adjacent 
characters, if any. These fonts allow a majority of capital characters to be represented in 5 pixel 
columns or less. 

The readability of such small fonts is greatly increased by the use of either subplxel 
optimization or anti-aliasing, because they allows Infomiation about the extent to which a character 
shape covers a given pixel to be represented at more than just a binary representation at the whole 
pixel level. In feet, subplxel optimization can be considered a type of anti-aliasing because It like 
traditional anti-aliasing causes pixels that are partially covered by a font shape to have color values 
that vary as a function of the extent of such coverage. 

FIG. 58 and 59 illustrate that subpixel-optimized bitnnaps produced by the present 
invention can be represented as font outlines and/or font bitmaps. The font outline descriptions 
5802 contain a mathematical geometric description of the shapes of one or more characters In a 
given font, preferably with hinting information designed to optimally place the boundaries of 
character outlines at one or more difforent font sizes. These font outlines can be ones, such as 
those just discussed, that have been designed to be rendered optimally on a subpixel addressable 
display and/or have hinting which is been optimized for display on a subpbcel addressable display. 

A font renderer 5806 can be used to create a subpixel-optlmlzed bitmap 5804 from such 
outlines, as is be described below. 

In some embodiments of the invention, illustrated in FIG. 58, a computer 5808 and/or an 
application which is running on that computer display text using font bitmaps or font outlines 
accessed over a computer network 5814 from a font server 5812. in other embodiments, 
illustrated in FIG. 59, a computer 5900 and/or an application 5902 ninning on it have font bitmaps 
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5804 necessary to render text stored vrithin them. Such computers and/or applications can store 
only font bitmaps, or they can store scalable font outlines 5802 and render font bitmaps 5804 as 
needed at different sizes. 

The advantage of storing only font bitmaps is that it prevents the need to store font 
outlines and a font renderer on the computer 5900. it also prevents the need for the computation 
involved in font rendering. Furthennore, many font vendors are much more willing to allow font 
bitmaps to be relatively freely available over the Intemet then they are font outlines. 

The advantage of storing font outlines is that if one is interested in rendering fonts at a 
large variety of sizes, it is actually more efficient to store the code necessary for the font renderer 
and to sbre scalable font outline descriptions than it Is to store font bitmaps for all the different size 
characters. 

The advantage of receiving fonts from a font server as shown in FIG. 58 Is that it allows a 
client computer 5808, such as that represented in FIG. 58, to represent text in any one of a large 
number of different font, size, and character combinations by downloading such fonts as needed, 
without the need to store a large library of fonts. Preferabiy the client computer 5808 will cache a 
reasonable number of character-font bitmaps so that there is no need to communicate over the 
networtc 5814 every time it seeYs to display a string. 

FIG. 60 IS a highly simplified pseudocode description of an algorithm 6000 used by some 
embodiments of the aspect of ttie invention relating to producing a subpixel-optimized font bitmaps. 
This algorithm uses nonlinear color balancing of the type described above with regard to FIGS. 48 
and 49. Sucti a subpixel optimization algorithm is particuiariy optimal for use in the display a text 
characters, because the alignment of text outline boundaries with hole pixel boundaries is quite 
common in rasterized font shapes because of the use of hinting. 

The algorithm 6000 of FIG. 60 includes a loop 6002 which is perfomied for each pixel row 
in the rasterization of an individual character-font shape at a given pixel resolution. This loop 6002 
is comprised of three subioop's 6004, 6008, and 6020, which are sequentially perfonned for each 
pbcel now . 
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The loop 6004 is performed for each subpbcel in tlie pixel row for which the current iteration 
of the loop 6002 Is being performed. For each such subpixel. the loop 6004 performs a function 
6006, which detemnines the coverage value for each such subpbcel as a function of the percent of 
the subpbcers area covered by the character-font shape of which an image is being made. 

FIGS. 61 through 90 are used to discuss methods which can be used to detennlne the 
coverage value of each subpixel in step 6006 of FIG. 60. 

As Is indicated in FIG. 61 , 62, and 63 the area in the image of the character-font shape for 
which such a coverage value Is calculated for a given pixel 5504 comesponds to the area of that 
image which will be displayed by each red, green, and blue subpixel 5506, 5508, and 5510, 
respectively. This is different than In the case of subplxel-optimized multicolor Images. In which the 
source image window con^sponding to each subpbcel Is larger, as is indicated in FIGS. 14 through 
16 above. The source Image window used by the method of FIG. 60 has the same sizes as the 
area of the source Image window used for bicolor bitmaps described above regard to FIGS. 42 
through 44. 

Such a higher resolution source image window can be used because the character-font 
shapes described by most font oufline descriptions are bicolor Images, with the area covered by 
the font outline considered as being associated with a foreground color (in most cases, 
represented by an alpha value of one) and all other portions of the Image being associated with a 
background cofor (In most cases, represented by an alpha value of zero). 

The calculation of the coverage values in function 6006 of FIG. 60 can be perfomied using 
any prior art technique capable of rasterizing a character font outline relative to an array of pbcels 
having the same spatial resolution as the subpbcePs of the grid 5502 shown In FIG. 55. 

FIG. 64 through 67 illustrate some of the traditional methods which a been used to 
calculate ttie percentage of a unit in a rasterization grid which Is covered by a font outline 6402. In 
the prior art, the unit of rasterization 6400 has typically been an anead comesponding to a whole 
pbcel in the output Image. In the method of FIG. 60 It is an area conesponding to a subpixel In the 
output Image. 
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FIG. 64 illustratBS one method of determining the coverage of a rasterization unit 6400 
wliidi uses matliematlcai techniques to exactly calculate the area of the unit that is covered by the 
outline 6402. This is lelatively computationaliy expensive, and thus Is hardly ever used. 

A sut)stantlaily more computationaliy efficient method is shown in FIG. 65, vidilch 
calculates the percentage of the rasterization unit 6400 that is covered by the outline 6402 by 
using piecevinse linear approximations 6504 of the boundary of the character-font shape. 

FIG. 66 illustrates an even more computationally efficient manner, although it produces a 
substantially less accurate results. This method detennines the percent of coverage of the 
rasterization unit 6400 by determining what percent of a set of sample points 6600 fall inside the 
shape of the outline 6402. 

FIG. 67 illustrates a method of detemnining coverage values which provides more accurate 
results far the same, relatively low degree of computation as the method of FIG. 66. It determines 
the coverage of the rasterization unit as a function of ttie average percentage of a number of scan 
lines 6700 and 6702 which are covered by the outline 6402. 

FIG. 68 through 87 illustrate an extremely computationally efficient method of calculating 
the coverage of a rasterization unit, which yields results that are typically better than a sampling 
method such as that shown in FIG. 66 for the same amount of computation. 

An embodiment of this method is described in much more detail in a U.S. patent 
application filed In the name of one of the inventors of the present application, Sampo J. Kaaslia. 
TOs U.S. Patent application has the serial number 09/363,513. It was filed on July 29, 1999, and 
Is entitled "Systems For F^pidly Performing Scan Conversion With Anti-Aliasing Upon Outline 
Fonts And Other Graphic Elements". This application has had Its disclosure published in PCT 
application PGT/USOO/21559. This application is incorporated herein by reference In its entirety. 

In the method of FIG. 68 through 87, the coverage value for a rasterization unit is 
detemilned by that percentage of one of ite two scan lines, a horizontal scan line 6804 or a vertical 
scan line 6802, that is covered by a font outline's shape 6402. TTie scan line whose coverage 
value is used as the coverage value for tiie rasterization unit is that which has the more 
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Intermedlate coverage value. For example, In an embodiment where the coverage for the 
horizontal and vertical scan lines is calculated in a range of values from 0 to 126, the scan line 
chosen Is that whose value is closest to 63, which represents a 50 percent coverage. 

In FIG. 68 through 71 K Is the percentage of coverage of the vertical scan line 6802 that is 
used to represent the percentage of coverage of the rasterization unit 6400. In FIGS. 72 through 
75 it is the horizontal scan line 6804 that has the most intermediate values, and, thus, which has its 
percentage of coverage used to represent the percentage of coverage of the entire rasterization 
unit 

In ail the rest of the FIGS. 76 through 87 it can be seen that the coverage value of the 
scan line with the more intennediaries coverage value nomialiy Is very close to the coverage value 
for the entire rasterization unit, and that it normaDy never varies fioom the coverage value of the 
entire rasterization unit by more than 25 percent 

FIGS. 88 through 90 represents other combinations of scan lines which can be used 
according to a method which we^hs the contribution of the coverage values of individual scan 
lines to the coverage value of their assodated rasterization unit as a function of which of those 
coverage values have more intennediate coverage values, in such methods the coverage value 
calculated for entire rasterization unit can be set equal to the sum of the coverage value of each 
scan line times its mediaiity, all dMded by the sum of each scans line's mediality. In this 
calculation, a scan line's mediality equals the scan line's middlemost percentage coverage value 
minus the absolute value of the difference between that middlemost percentage coverage value 
and the scan line's actual percentage coverage value. 

FIG. 91 illustrates a hypothetical font outline 9102 mapped over the red, green, and blue 
subptol's 5506, 5508, and 5510, respectively, of a row 9100 of pixels 5504. 

FIG. 92 illustrates the corresponding coverage values 9202 which have been calculated 
for each of the subpixel's all the row 9100. 
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FIG. 93 illustrates how the coverage values determined for an individual subpixel can be 
distribuied using a linear color balance method. This linear color balancing is identical to that 
described above with regard FIG. 46. 

Returning briefly now to FIG. 60, once step 6006 of that figure has calculated or estimated 
the coverage value for each subpbcel of a row, as indicated in FIG. 92, a loop 6008 is perfonDed for 
each pixel In the row. This loop color balances the coverage values calculated for the subpixels of 
a row. It does not use a linear color balancing routine of the type illustrated in FIG. 93 and 
described above with regard to FIGS. 46 and 47. Instead it achieves higher perceivable spatial 
resolution by using a non-linear color balancing technique similar to that described above with 
regard to FIGS. 48 through 53. 

The loop 6008 perfonns two func^ns, 6010 and 6012, and a loop 6014 for each such 

pixel. 

The function 6010 finds which subpixel of the cunent pixel has had the minimum coverage 
value calculated for its subpixel. Then step 6012 adds this minimum coverage value to the 
temporary alpha, or opacity, value being calculated for each subpixel of the cument pixel 

Then a loop 6014 performs function 6016 and 6018 for each subpixel of the cun^ent pixel. 
The function 6016 determines, for the current subpixel of the loop 6014, the excess of the 
coverage value which has been calculated for it over the minimum coverage value which has been 
found for the pixel of which the subpixel is part Then function 6018 distributes this excess value 
across the subpixel alpha values being calculated for the cuo'ent subpixel and the two subpixels to 
its left, and the two subpixels to its right in the cunent pixel row. This function conesponds to that 
described above with regard to FIG. 49. 

FIGS. 94 and 95 illustrate two different color balance distribution filters which are used in 
one embodiment of the present invention. In this embodiment an asymmetrical center weighted 
color balance filter shown in FIG. 94 Is used to distribute the coverage values associated with the 
red and green subpixels. The asymmetrical color balance filter shown in figure 95 is used to 
distribute coverage values associated with blue subpixels. Thus, this embodiment of the invention 
differs from the process described above v\^th regard to FIG. 49 in that it used differently shaped 
distribution filters for some colors than for others. 
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One of the inventors of the present application has found that because the eye perceives 
green much more strongly that it does blue, that color balancing coverage values associated With 
differently colored subpbcels should use such different distribution filters. In other embodiments of 
the invention relating to non-linear color balancing (including the non-linear color balancing of bi- 
oolor images) a different color balancing filter could be used for each different color, the same color 
balance filter could be used for all colorSi and either symmetrical or asymmetrical color balancing 
filters can be used. 

The particular color-balancing filters shown in FIGS. 94 and 95 are designed for use with 
coverage values that are calculated on a scale from 0 to 126. A given coverage value having a 
value from 0 to 126 is associated with the set of five distribution values on the right hand side of 
the tables of FIGS. 94 and 95 whose associated color value on the left side of that table is closest 
to Its own color value. For example, if the coverage value of the cument subpixel was 126 for the 
colors rad or blue, an addition of 1 would be made to the alpha value being calculated for subpixels 
two to the left and two to the right of the cunent subpbcel, an addition of 3 would be made to the 
alpha values being calculated for the subpixels one to the left and one to the right of the cunent 
subpixel, and a value of 4 would be added to the alpha value being calculated for the cunent 
subpixel. in this particular embodiment the alpha values are calculated on a scale from 0 to 12. 

The relative size of the color balance distribution shown in the last row of FIGS. 94 and 95 
reflect more accurately the desired distribution ratios. This Is because the larger value distributed 
in each of these last rows allows greater numerical resolution than is found in the rows above each 
of them. 

It should be appreciated that in other embodiments that use higher numerical accuracy to 
describe the coverage or luminance values being balanced, the balancing distributions would have 
ratios between the contributions to different subpixels more lil^e those reflected In these last rovi^ 
of FIG. 94 and/or like those shown in the last row of FIG, 95. This is particulariy true when filters of 
the general type shown in FIG. 94 and/or FiG. 95 are in the color balancing of bicolor subpixel 
optimizations of images, such as is described above with regard to FIGS. 48 through 52, This Is 
because, in such bicolor subpixel optimizations of bitmap Images, there is little benefit in computing 
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the lurrdnance to be color balanced at a resolution lower than that used the bitmap being subixel 
optimized. 

Once loop 6008 of FIG. 60 has caused step 6018 to be perfomned for each subpixel of 
each pbcei in a row, each pixel will have a separate alpha value calculated for each of Its three 
subplxels, with each such alpha value having one of thirteen opacity levels. This means It Is 
possible for each pixel to have 1 of 2.197 (i.e.. 13^) different possible combined alpha values. In 
other embodiments of the invention alpha values v\dth higher or lower resolution can be used. 

In many embodiments of the invention, particularly those designed to run on computers 
vwth limited computational capacity or in systems in which it is desirable to reduce the bandwidth or 
storage capacity required to store or download font bitmaps, it Is desirable to map from the 
relatively large color space of ttie 2,197 combination of different subpixel alpha values possible 
after such color balancing into a smaller color space. 

The embodiment of the invention in FIG, 60 performs such a mapping. Once the loop 6008 
has been performed for each pixel in the cunent row, a loop 6020 perfomis an additional function 
6022 for each such pixel. The function 6022 takes the three alpha values which have been 
calculated for each of a pixePs subpixeis and uses them as an input value of a lookup table that 
maps from each of the 2,197 possible color value defined by the possible combinatfon of a pixel's 
three alpha values Into 1 of 122 values, in this embodiment the color space has been reduced 
down to such a small number of colors so that a machine that has a 256 value color space will be 
able to display each of the 122 values selected for use in the display of subpbcel optimize fonts 
while still having over half of such a limited color space for other uses. The uses of such a small 
color palette to represent font bitmaps reduces the number of bits required to store such font 
bitmaps and makes them more efficient to download, in other embodiments of this aspect of the 
invention the source and the destination color spaces used in such a mapping could have different 
sizes. 

FIG. 96 illustrates the method 9600 which has been used to create such a color mapping 
in one embodiment of the prefened invention. It is to be understood that in other embodiments, 
other types of mapping could be used, and as indicated above. In some embodiments no such 
mapping Into a smaller color space need be used at all. 
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The method of FIG. 96 starts with a step 9602 which runs multiple characters from multiple 
fonts through the non-linear color balance subplxel optimize algorithm described above with regard 
to FIGS. 60 through 95. When this Is done a histogram is kept of the number of times each of the 
possible 2.196 different pbcel alpha values Is calculated for any of the pixels. This histogram is 
useful because most of the three-colored alpha values calculated for pixels in subpixei-optimized 
font bitmaps tend to be concentrated into a various small regions of the total possible color space 
of 2.196 such three-color alpha values. This concentration is probably even more pronounced with 
non-linear color balancing, because it substantially reduces the amount luminosity distributions due 
to color balancing. 

Next a function 9604 creates a limited color palette, in this case having 122 colors, by 
perionning the functions 9606 and 9608. The function 9606 selects the thirteen grayscale values 
which are possible for whole pixel alpha values, given that each subpixel can have one of thirteen 
alpha levels. Then the function 9608 selects the 109 other most frequently occunlng colors in the 
histogram previously calculated by step 9602. 

Once the limited color palette has been selected, a loop 9610 Is perfomned for each of the 
2,196 possible whole pixel alpha values. For each such possible alpha value a conditional 9612 
tests to see if that input color exactly matches one of the 122 colors. If so, the funcflon 9614 
associates the input color with its identical output color in the lookup table being constmcted. If the 
condition 9612 is not met, a loop 9618 and a function 9628 will be performed for the cunrent input 
color of loop 9610. 

The loop 9618 is perfonned for each of the 122 output colors in the palette. It has a 
conditional 9620. which tests to see if the difference between the red alpha value of the input color 
to be mapped and the current output color of the loop 9618 is of the same sign as the difference 
between the green alpha value of the cunent input color and the green output alpha value for the 
cunent output color. The conditional 9620 also tests to see if the difference between the red alpha 
value and the green alpha value of the current output color is less than the difference between the 
red alpha value and the green alpha value of the input color (plus a possible value X to allow some 
leeway). If these two conditions, which are designed to prevent relatively noticeable differences 



wo 02/088979 



PCT/US02/14380 



-54^ 

between an input color and the output color to which it is to be mapped, are met, functions 9622 
through 9626 will be perfonfned. 

Function 9622 calculates the distance from the input color to the output color. Function 
9624 tests to see if that distance is the closest distance so far to the input color in the current loop 
9618. If the test of function 9624 is meet step 9626 saves the current output color of the loop 
9618 as the closest allowed palette color. After the loop 9618 has been perfomied for each of the 
122 output colors of the limited palette, step 9628 associates the cument Input color of the loop 
9610 with the closest allowed palette color calculated in ttie loop 9618. 

Once the loop 9610 has been perfonned for each of the possible input colors, each of 
those input colors will have been mapped to one of the 122 output colors. 

In the particular cx)lor mapping scheme shown in FIG. 96 non-grey scale pixel color values 
produced by color balancing get mapped in to greyscate color values if they do not get mapped 
into one of the one hundred and nine most frequently occuning non-greyscale color values 
selected by step 9608. This generally yields results at least as good as traditional anti-aliasing, 
which represents ail bitmaps with a greyscale alpha value. 

FIG. 97 illustrates an algorithm 9700 used to display font bitmaps of a type generated by 
the methods of FIG. 60 and 96 on a subpbcel addressable screen. 

The loop 9702, comprised of the function 9704 any loops 9706 and 9714, is perfonned for 
each string to be displayed. 

Function 9704 samples a set of points in the rectangle of the bitmap at which the string is 
to be drawn, to detemnine the average bacl^ground color value for the string. In other 
embodiments ttie background color is separately detemilned for each character or for each pixel of 
each character, but in the embodiment shown, the background color is determined only once for 
each string to save computation. 

Once the background color for the string has been detemilned, loop 9706 perfonris a 
subloop 9708 and a function 9712 for each of the 122 whole pixel alpha values, described above 
with regard to FIG. 96. 
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The toop 9708 performs a funcBon 9710 for each of the three subplxel colors. The 
function 9710 calculates the luminosity value for the current subpixel color as a function of the 
components of the current whole pixel alpha value con'espondmg to that cument subpixel color, it 
sets the luminosity value it Is calculating equal to this subpixel alpha value multiplied by the 
luminosity of the cunent subplxel's conesponding color In the foreground color of the string to be 
drawn, plus a quantity of one minus the cunpent subpixePs alpha value multiplied by the luminosity 
of the cunent subpb^Fs corresponding color in the baclcground color determined by function 9704. 

Once this loop has been perfomned for each of the three subpixel colors, function 9712 
maps the cunoent whole pixel alpha value of the loop 9706 into the whole pixel color value 
comprised of the three subpixel luminosities which have Just been calculated in the loop 9708. 

Then the loop 9714 performs the function 9716 and the loop 9718 for each of the 
characters of the cunent string b be displayed on a subpixel addressable display. 

Function 9716 accesses the font bitmaps for the current diaracter. Then the loop 9718 
perfomns functions 9720 and 9722 for each pixel of that bitmap. Function 9720 finds the color 
value which has been mapped by the loop 9706 Into the cunent whole pixel alpha value indicated 
for the cunent pbcel In the character's font bitmap. Once this color value has been found, function 
9722 sets the conesponding pixel in the subpbcei addressable display to the that whole pbcel color 
value. 

Once the loop 9718 has been perfbnned for each pixel of each character of the string, the 
string will have been completely displayed in a subpbcei optimize manner. 

FIGS. 98 through 101 are used to illustrate how well the techniques for image and font 
scaling and subpbcei optimization woric. FIGS. 98 and 100 illustrate views of two different web 
pages laid out and displayed at 640 by 480 pixels using a common browser program. FIGS. 99 
and 101 Illustrate the same web pages after their images and text have been scaled by the method 
described above so as to fit on a 320 by 240 display. Unfortunately, the 320 by 240 pixel images 
are printed with grayscale values detennlned by the average luminosity of its whole pbcels, and 
ttius the actual clarity added by subpixel resolution is not shown in these Images. 
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FIGS. 102 through 113 Illustrate In more detail the Interaction between a proxy server and 
a thin client computer in one emt}odiment of the present invention. 

FIG. 102 is a highly schematic box diagram of a system Including a proxy server 210 and a 
thin client 200 of the type described above in regard to FIG. 2. 

The proxy server 210 includes a browser 10200 which includes programming 10202 to 
perform the standard functions of a full Web browser. This programming has been modified 
because the browser operates as a proxy lor the thin client When the browser receives over the 
network an HTML description 10204 of a requested web page, it creates a two dimensional layout 
10206 of that web page. 

FIG. 103 illustrates a portion of HTIWL description of the web page whose display is shown 
in FIGS. 98 and 99. The numerals 10300 shown in FIG. 103 illustrates portions of text in Sie 
HTML which are shown In the left-hand column of the web page shown in FIGS. 98 and 99. The 
numeral 10302 points to an Image tag that identifies the bitmap used to represent the word 
"Sections" shown in the same column. 

When the browser code receh/es the download of the web page, it attempts to create a 
layout 10206 of that web page at a virtual screen resolution, which conosponds to the size of the 
window into which it thinl^s It is displaying ail or a portion of the web page. We call this window into 
which the browser thinlcs it is displaying the web page fte wrtual screen 10208. 

FIG. 104 illustrates the layout 10206 of the web page shown in FIGS. 98 and 99, and it 
shows in heavy black rectangle 10208 the mapping of the virtual screen into that layout. 10220 
shows the actual screen image which is displayed on the thin client given the location of the virtual 
screen shown in FIG. 104. 

Many web pages today include elements larger than the 640 by 480 virtual screen 
resolution used in the example system being described. The layout will have the minimum width 
required to layout the objects of the web page, or the width of the virtual screen, which ever is 
larger. For example, It is common today for many web pages to be laid out with a minimum 
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posslble resolution of 800 pixels. In this case the virtual screen will have a smaller width than the 
layout This Is the case In the example shown in FIG. 104. The view window 10210 shown In FIG. 
102 represents that portion of the virtual screen which is to be actually displayed upon the screen 
of the thin client In views shown In FIGS. 99 and 101 the view window equals the virtual screen. 
But as the user zooms in on a portion of the virtual screen, the zoom's scale Victor control 10216 
will change and the view window will be mapped Into a subset of the virtual screen. 

Scroll control 10218 causes the view window to move relative to the layout if the view 
window is nrioved so that it indudes a portion of the layout which is not on the virtual screen, a 
command will be sent to the browser software to scroll ttie virtual screen. The event queue 10220 
stores events, that is, user input which have been received on the thin client and which have been 
uploaded to the proxy server for conresponding action by the browser. Events which occur on the 
screen of the thin client are mapped through the view window to the corresponding locations on the 
virtual screen and then placed in the event queue of the browser, so that the browser will respond 
to such input as if it had been received at the appropriate location on the screen (aica the virtual 
screen) which it thinks it is drawing directly onto a video output device. 

The browser programming 10202 of FIG. 102 has been modified so that each time it thinks 
it is drawing an object on the virtual screen It creates a conesponding scaied<iown object at a 
conespondingly scaled location in a download display list 10212. 

This display list is downloaded over the network 10222 to the client computer, which stores 
it as Is Indicated by the numeral 10212A. The scaled down images refened to by this display list 
10214 are also downloaded. Programming 10218 located on the thin client displays the strings, 
images, and other elements contained in the display list on the thin client screen 10221 . if the user 
clicks on the thin client screen, the operating system 10222 of the thin client places such a click 
and its location on the thin client's screen in an event queue 10224. Each such event which does 
not relate to programming handled tocally on the thin client is uploaded to the event queue 10220 
of the proxy server, as described above. 

FIGS. 105A through 110 are highly simplified pseudo code descriptions of programming 
and data structures on the browser and thin client computers designed to control their interaction 
for the purpose of allowing the thin client to browse web pages through the proxy. 
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FIGS. 105A and 105B are highly simplified pseudocode representation 105000s of 
portions the brov\ser'8 oode 10202 shown in FIG. 102 used to help it function as a proxy for the 
thin client 

In the particular embodiment illustrated in these figures, a lange Web browser designed for 
normal use has been patched so as to make it perfonn as a proxy. It is to be understood that in 
other embodiments of this aspect of the inventton the functionality necessary to malce the browser 
operate as a proxy could be nione intimately and elegantly integrated into the browser's code. In 
yet other embodiments, code In the operating system, or in functions which intercept operating 
system calls can be used to malce a standard Web browsing program operate as a proxy for a thin 
cItenL 

In the embodiment shown in FIG. 105A, if the browser receives a request from the thin 
client for a web page, steps 10502 and 10504 relay that request to the server computer indicated 
in the URL of the request. 

if the browser receives an indication from its own code that the browser has completed a 
draw or redraw of the virtual screen 10208 described above with regard FIG. 102, functions 10506 
and 10510 will call the screen capture and download routine shown in FIGS. 106A and 106C. 

FIGS. 106A through 106C are highly simplified pseudo code descriptions of the screen 
capture and download routine 10600. 

When this routine is called by function 10510, just described, its step 10602 asks the 
browser for a screen redraw, which causes the browser to call routines to draw each of the 
elements in the web pages layout which all or partially fit within the virtual screen. The routine of 
FIGS. 106A through 106B records infonnatlon contained in each of these draw calls and uses It to 
create the download display list 10212 shown in FIG. 102. 
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If the browser calls a measure string routine 10606 of FIG. 106A, this routine causes 
functions 10608 through 10618 to be perfonmed. Such calls are made by the browser to determine 
the size of text it Is seeking to layout Into the virtual screen. Although not shown in the figures, 
these same functions 10608 and 10810 are perfomied anytime the browser makes a call to 
measure string size, even if It Is not during the operatton of the screen capture and download 
routine shown in FIGS. 106A through 106B. 

Function 10608 maps the font specified in the measure string call into a font having a 
different font family and a different font size. This font substitution is controlled by three 
considerations indicated by numerals 10608 through 10616. 

Consideration 10608 seeks to select a size for the substitute font as a function of the 
requested font size in the call to the measure string routine and the display scale factor. 

The display scale factor Is a ratio of the resolutton along a given dimension of the portion 
of the virtual screen 1028 conBsponding to the view window and the resolution, along the same 
dimension, at which the view window will be displayed on the thin client In some cases the display 
scale factor will have different components to represent different scaling ratios to be used along the 
horizontal and vertical directtons, but in many cases the display scale factor will be comprised of a 
single scaling ration to be used for both horizontal and verfical resolution. 

in the embodiment shown In FIG. 102. this scale factor is stored In the Zoonn/Scale Factor 
Control 10216.. In cases where the virtual screen has a resolution of 640 by 480, the view window 
equals the size of the virtual screen, and the view window Is displayed on all of a 320 by 240 
display, the display scale factor will be two, nfieaning that elements are to be drawn on the screen 
of the thin client at 1/2 the pixel resolution at which the browser thinks it is drawing them upon Its 
virtual screen. 

Consideration 10612 replaces ail font sizes which will be small when displayed on the thin 
client screen with font families which are narrower and taller than the average pfacel size of the font 
which would be selected by the consideration 10610 alone. When reducing from a 640 x 480 
virtual screen to a 320 x 240 display screen this can include most or all web page text represented 
in character, as opposed to bitmap, fomi. This substitution is done because the subpixel 
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addressable displays used with this embodiment of the invention have three times the subptxe! 
resolution In the horizontal direction as they do in the vertical direction. Because of this, 
decreasing the width of characters has a less negative Impact on readability than decreasing the 
their height. Thus, to display the maximum amount of relatively easily readable text on such a 
subpixel addressable display screen, this substitution caused thewidth of characters to effectively 
be scaled down by more than the display scale fector and the height of such characters to 
effectively be scaled down by less than the display scale factor. For example, the fonte of the 
small screen displays shown in FIGS. 56, 57, and 99, 101, 168, 169, 172, 173, and 174 have all 
been substituted by fonts which have been scaled in such a manner. 

The fonts in these figures have a pixel size of eight pixel per em. A majority of the lower 
case letters in this font fit within an advance width of four pbcel columns of less. This width of four 
pixel columns or less includes the spacing, if any, that occurs between the shapes of characters 
having such widths. In these particular fonts, over eighty-percent the lower case characters of the 
roman alphabet fit within such an advance width These characters have an x-hejght of more than 
four pbcel rows, which malces them generally considerably taller than they are wide. As a generally 
rule, such a relatively narrow font can represent a larger amount of text within a given area at a 
given level of readability than a wider font 

The consideration represented by the numerals 10614 and 10616 tests to see if a flag has 
been set to limit minimum font size, indicating that no fonts should be shown on the thin clienf s 
display below a certain pixel size. Commonly this flag will be set to prevent the display of text that 
is too small to read. It can be unset when the user desires to see a more accurate scaled-down 
representation of how the web page text would nomially be laid out if actually shown on a display 
having the virtual screen size. Such a desire is particularly likely when the display scale factor is 
large, meaning that placing such a minimum limit on text size would drastically alter the 
appearance of the web page's layout 

If, as is often the case, the system is limiting minimum font size, then steps 10614 and 
10616 prevent the substitute font size from being below a minimum pixel size. In a cunent 
embodiment of the invention, this minimum pixel size is eight pixels per em. The developers of this 
embodiment developed hinted fonts for subpbcel display at seven pixel per em, and although they 
found such fonts relatively easy to read, they received feedback from other users that such small 
fonts were too difficult to read. 
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The limitation on minimum font size often substantially changes the relative size of fonts at 
which a web page's variously sized fonts are actually displayed. 

In some embodiments of the invention, all Web text is displayed at one font size. This 
actually worte quite well for most web pages, because in most web pages the tally large fonts are 
represented by bitmaps. 

Once the function 10608 has detennlned which font family and font size should be 
substituted for the font with which the measure string routine has been called, function 10618 
returns the string measurement of the string with which the routine was called, given the size of the 
string's characters In the substituted font and font size, after that measurement has been scaled up 
by the display scale factor. 

The return of this value causes the browsei's layout engine to lay out the web page using 
font metrics for characters that are scaled, relative to the pbcel size at which those characters vnW 
actually be displays by the display scale factor, which is the ratio of the resolution of the portion of 
the virtual screen conesponding to the view window and the actual resolution at which the view 
window will be displayed on the thin client screen. This means that the virtual screen is being laid 
out using font metrics that are different than the actual font metrics that will be displayed as a result 
of that layout 

If the screen capture and download routine receives a call to a string draw routine 10620, 
this routine causes functions 10621 and 10624 to be perfonned. 

Function 10621 transforms the sc^en position at which the string Is to start being drawn 
Into the conesponding position on the thin client screen at which the string will ultimately be 
displayed. This transfonnation takes into account the mapping between the view window 10210 
and the virtual screen 10208 illustrated In FIG. 102. This mapping reflects both the cunent zoom 
setting stored by the control 10216 and a cunrent scroll setting stoned by the scroll control 10218 
also shown in FIG. 102. 



wo 02/088979 



PCTAJS02/14380 



-62- 

Function 10622 tests to see if the sut)stituted font family and size assodated with the 
string by the prior call to the measure string routine, described above with regard to numerals 
10606 through 10618, and any other font attributes requested for the display of the cunrent string, 
are different than the cunent values for such font attributes. The cunent value for each such font 
attribute is defined by the last value for each such attribute defined by a font commands already 
recorded In the download display list If such differences are found, function 10623 stores a font 
commands at the current end of the display list changing any such font attributes to 
thoseappropriate for the display of the current string. 

Function 10624 stores the string with which the string draw routine has been called and 
the transfomned screen position just calculated by step 10622 at the end of the downtoad display 
list 10212. illustraled In FIG. 102. As described below with regard FIG. 108, this is done by placing 
a string command in tiie display list containing the string's transfbnned start position and its 
characters 

If the screen capture and download routine receives a call to a rectangle draw routine 
10626. this routine causes functions 10628 through 10634 to be peitomed. Rectangle draw 
commands are commonly called by browsers to create areas of a web page with different 
background color, as well as to draw horizontal and vertical fines which can be used as underiining 
for text or demarcations between different portions of the web page's la/out. 

Function 10628 transfonns the geometric values contained in tiie call to the 
conesponding geometric values with which a conesponding rectangle will be drawn on the tfiln 
client's display. This Includes transfonning the rectangle's start- screen position, and its width and 
its height 

Function 10630 tests to see if the rectangle's color is different tiian ttie cument (I.e., last) 
rectangle color in tiie display list If so, function 10632 adds a background color command to the 
end of the display list changing ttie cunient background cotor to ttie color specified in the cunent 
call to tiie rectangle draw routine. 

Next function 10634 stores the rectangle and Its transfonned screen position, widtti. and 
height at tiie end of tiie download display fist witfi a rectangle command. 
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If the screen capture and download routine receives a call to a bitmap draw routine 10636 
shown in FIG. 106B, this routine causes functions 10638 through 10670 to be performed. Bitmap 
draw routines are called by browsers to display pictureSi pictures of fonts, banner ads, and Images 
associated with hot zones and other graphical user Interface bitmaps of a page. 

In some embodiments, only the first screen of given animations are captured and 
recorded to the download display list to reduce the anfK>unt of bandwidth required to display web 
pages. In other embodiments, particularly those with higher bandwidth Ms such a restriction 
need not apply 

In the embodiment of the invention which is described with regard to FIGS. 106A through 
160C, bitmap draws associated with certain graphical user interface's are ignored because the thin 
client's programming stores subpbcel-optimlzed, scaled-down bitmaps for such controls. 

Step 10638 tests to see If the URL of the image for which the bitmap draw routine has 
been called Is already In a download Image fist, not shown in the figures, which contains each of 
the images referred to In the download display list. If not, the requested bitmap has not yet been 
processed for the current download and functions 10642 through 10662 need to be perfomned for 
it. 

Function 10642 tests to see if the bitmap is a cofor bitmap. If so it causes functions 10644 
through 10654 to be perfomied. Function 10644 scans the color images for one or more individual 
areas of sufficient size to Justify separate treatment which each contain only colors fifom a single 
bicolor spectmm. A bicolor spectrum oomesponds to a set of colors which lie In a line in an RGB 
color cube (l.e. a color cube defined by red, green, and blue value ranges In each of Its three major 
dimensions). 

For each bicofor portion of the image found which is large enough to Justify individual 
processing, function 10646 causes functions 10648 and 10650 to be perfomfied. FuncOon 10648 
perfomns a bicolor subpixel optimization, of the type described above with regard to FIGS. 42 
through 53, on the cunent portion of the image using the most extreme ends of its bicolor spectmm 
as its foreground and baci^round colors, and using the current display scale factor to detenmlne 
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the extent to which it scales down that portion of the image. This subpixel optimization, lilce that 
performed in steps 10654 and 10658 described in the next few paragraphs, scales down the image 
by the display scale factor, which is the ratio between the resolution of the image in the virtual 
layout of the proxy browser and the resolution at which it will be displayed on the thin client's 
screen. 

After this subpixel optimization has been perfonmed, function 10650 detennines If the 
foreground color is too chromaticatiy unbalanced. That is, it Is to close to a pure red, green, or blue 
cobr. If this Is the case, such color purity would decrease the accuracy with which it can display 
the spatial resolution of the color image. If this is the case, the foreground color can be replaced 
by a corresponding color which is closer to a grayscale value, and thus which will allow more 
accurate spatial repmsentaflon. 

In some embodiments of the invention such foreground color substitution will not be used 
because it might upset the color amounts of the color Image. In general It is best not to use such 
foreground color substitution unless the foreground color appears throughout a substantial portion 
of the entire color image. In other embodiments of the invention the background color associated 
with a bicolor image could be changed. But the Changing o the background colors of images on 
web pages is often unadvisable. 

For each non- bicolored portion of the cunent image, function 10652 causes step 10654 to 
perfonn a multicolored subpixel optimization, of the type described above with regard to FIGS. 14 
through 41, on that portion of the bitmap at the cunBnt display factor. 

If the bitmap for which the bitmap draw routine has been called is a grayscale bitmap, 
function 10656 causes step 10658 to perfonn a bicolor subpixel optimization on the bitmap using 
black and white as the foreground and background colors at the current display scale factor. 

Then function 10662 stores the scaled-down, subpixel-optimized bitmap at the end of the 
image list with a unique image ID, its URL, and its scaled width and height 

Whether or not the image with which the bitmap draw routine has been called was 
previously in the image list, by the time the program advances to function 10664 It will be in that 
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list, and will have been assigned an ID number and a transformed width and height At this time 
function 10664 transfomis the screen position with which the bitmap draw routine has been called 
for the Image to one applicable to the thin clients screen, and then stores an image location 
command of the type shown In FIG. 108 having the image's image ID, its transfomied screen 
position, and its transfbmried width and height at the end of the download display list. 

In some embodiments of the invention all bitmap images are subpbcel-optimized using the 
multicolor subpbcel optimization routine. In other embodiments only grayscale bitmaps undergo 
any bicolor subpixel optimization. 

In some embodiments of the invention vector Images can be handled by performing 
subpbcel optimization upon the shapes defined by such vector descriptions. In some such 
embodiments such subpbcel optimization Is performed on the proxy, but in others it is perfomrted on 
the thin client One of the advantages of vector, or geometrically defined, drawings it is the 
compactness with which their descriptions can represent an image. Thus when bandwidth to the 
thin client is a primary restriction, it might well make sense to download vector descriptions of 
images and have the thin client then render them using subpixel optimization. 

It is possible in some embodiments, to have image recognition performed upon images, 
and then have the recognized images downloaded to the thin client in a symbolic representation. 
For example, it is common in many web pages to represent large text with bitmaps. Optical 
character recognition could be perfonned on such bitmaps, and corresponding characters and their 
font, or an approximation of their font could be downfoaded symbolically, so as to reduce the 
bandwidth required in order to descnl)e the page to the thin client 

If the screen capture and download routine receh^es a call to the routine to create a control 
object, such as a radio button, check box, text field, or button from the browser, the controlCreate 
routine 10866 shown In FIG. 106C causes functions 10667 through 10670 to be perfonned. 
Function 10667 transfomis the screen position at which the browser has requested a control to be 
drawn to the location at which it Is to be drawn in the thin client's screen. A function 6668 places a 
con-esponding control create cowrmi as indicated in FIG. 108 in the download display list, 
including its corresponding text label, and function 10670 creates a corresponding browserside 
portion of the control object 
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In this embodiment of the invention the functionality of a control object shown in the thin 
dienf s screen is shared between the proxy and the thin ciienL State infomrtation, such as whether 
not a chedc boxe is checlced, or which set of radio buttons has been pushed, is stored on the thin 
cilent. This prevents the need for communication from the thin client to the proxy every time the . 
user enters infomiatlon into such a control object Usually It is only when the user clicks some sort 
of a button which indicates that the Infonnation stored for such controls is to be transmitted to the 
remote server computer which originally generated the web page, that the cilent needs to send 
such Infomoalion to the proxy, for relay to such a server. 

In other embodiments of the invention having a higher bandvridth link to the thin client, it 
might be desirable to simplify the code of the thin client, by having mors or substantially all of the 
functionality associated with indhridual control objects mn on the proxy. 

When the screen capture and downtoad routine determines that the screen redraw 
requested by function 10602 of FIG. 106A is complete, function 10672 of FIG. 106C causes 
function 10764 to call the download display list routine 10700 shown in FIG. 107. 

As shown in FIG. 107, the download display list routine has a fiincSon 10702 that places 
all elements in the download display list which are to be totally or partially newly displayed on the 
new thin client's screen in a download stream. Nomially this includes any elements in the 
browser's vbtual screen which occur within the cunent view window. As is explained below, 
however, in the case of a scroll In which a significant portion of the prior bitmap on the thin cl!ent!s 
screen can be reused, only elements which occur at least partially in the portion of the view 
vAndovi that is not in the reusable portion of the thin client screen's cunent bitmap are placed in the 
download stream. 

In many embodiments of the Invention the functions of FIGS. 108A through 106G which 
creates the download display list do not enter an element on the download display list if it does not 
fit within the view window. In other embodiments this filtering takes place in function 107002. 

In some embodiments of the invention elements which are downloaded are clipped, so 
that only those portions of such elements which are to actually fit within the thin client screen are 
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downloaded. This would have the benefit of decreasing the number of bits required for download, 
but it would add computational complexity. 

Once all the elements on the download display list to be shown on the thin client screen 
have been placed in the download stream, function 10704 places the bitmaps of all images with a 
conpesponding image location command in the download stream at the end of the download 
stream, as indicated by the numeral entries 10818 in FIG. 108. Some embodiments of the 
invention, before they places such bitmaps at the end of the download stream perfomi a lossy 
compression on them. In some embodiments, the algorithm used Is one that clusters the color 
\^ues in the image into clusters of colors having visually imperceptible differences in RGB color 
values, using a metric which talces into account the fact that green color values differences are 
more perceptible than red color value differences, and that red color value differences are more 
perceptible than blue color value differences. 

Then fundion 10705 compresses the download stream, Including the Images previously 
compressed by the lossy algorithm, using a lossless compression algorithm. Standard prior art 
lossless compression algorithms can be used for this purpose. 

FIG. 108 is a schematic illustration of such a download display stream. In some 
embodiments such a stream is actually represented using a maricup language. 

The font commands 10812 shown In FIG. 108 represent font commands recorded in the 
display list by function 10623 of FIG. 106A. 

The string commands 10814 of FIG. 108 represent commands recorded in the download 
display list by the step 10624 of FIG. 106A. 

The background color commands 10806 of FIG. 108 represent the bacl(greund color 
commands entered by the function 10632 shown In FIG. 106A. 

The rectangle commands 10808 of FIG. 108 represent rectangle information stored by 
function 10634 of FIG. 106A. 
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The image location commands 10810 shown in FIG. 108 repfBsent image location 
commands recorded by the function 10664 of FIG. 106B. 

The control commands 10816 ofh FIG. 108 represent control commands placed In the 
download display list by the function 10668 of FIG. 106C. 

Returning now to FIG. 107, once all the elements for the download stmam have been 
selected and the stream is ready to be sent, function 10706 opens a soclcet connection between 
the browser computer and the thin client and then function 10708 sends the download stream's 
display list infomnation down to the thin client The thin client then displays infonnatbn, as is 
described below in greater detail with regard to FIGS. 109A flirough 109C. 

Returning now to FIG. 106G, once the call in the function 10674 to the download display 
list routine is complete the function 10676 dears the display list so the new display list can be 
created for the next screen which is to be downloaded to the thin client 

Returning now to FIG. 105A, we have just described the completion of the screen capture 
and download routine called by function 10510 shown in that figure. 

As shown in FIG. 105A, if the browser's proxy code receives a query from another portion 
of the bmser code for the state of one or moro control objects displayed on the thin clients 
screen, fundion 10516 sends a query to thin dient for the state of that one or moro control objeds. 
When such state infonnation is received from the thin dient it is rotumed to the programming 
which made the request for such state infonnation. 

As was described above with regard to functions 10666 through 10670 of FIG. 106C, this 
embodiment of the invention adualiy has the thin dient draw and store state infonnation about 
individual control objects, such as radio buttons, ched( boxes, and text entry fields, to reduce 
communication bandwidth as the user changes infonnation prior to selecting to have it submitted to 
the web site on whose web page such controls are shown. Commonly vA\er\ the user dlcl<s on a 
submit button the associated diclc event is transmitted up to the proxy computer, it has its screen 
coordinates transfomied the conesponding coordinates on the virtual layout screen, and then it is 
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placed In the browser's event que for the browser code to respond to that click event as if It had 
been generated on the screeni having the ^^rtual screens resolution, which the browser thinks it is 
displaying. Once this Is done, the browser code traditionally asks for the state of all of the current 
web page's control objects, so It can post that infonnation back to the web server from viriilch the 
cunBnt web page came. It Is such requests that cause the operation of functions 10514 through 
10518. 

If the browser's proxy code receives a scroll or nnove command from the thin client 
functions 10522 through 10534 of FIG. 105A are perfonned. 

Function 10522 moves the view window 10210 shown In FIG. 102 relative to the browser's 
layout 10206 In response to the scroll or move. Then fundion 10526 tests to see If any significant 
portton of the view window which was In the view window before the move Is still in the view 
window after the move. If this is the case, it means a substantial portion of tiie bitmap cunrentiy 
being displayed on the thin browser screen can be reused In ttie display after tiie requested scroll 
or move is accomplished. In tills case function 10528 places a scroll commands 10804. Illustrated 
near tiie top of tiie download stream In FIG. 108, at ttie start of tt)e new display list which Is to be 
created for Uie scrolled screen. Such a scroll commands includes an XY shift value whteh 
indicates which portion of tiie ttiin client's prior screen bitmap is to be reused. 

In FIG. 108 botii a ctear command 10802 and a scroll command 10804 are shown at tiie 
start of ttie downfoad stream, so ttiat botii can be Illustrated. In tiie current embodiment only one 
of tiiese two commands, tiie clear command or tiie scroll command will start a download screen, 
wttti ttie first being used if ttie screen of ttie ttiin client Is to be totally redrawn, and ttie second 
being used If a portion of ttie thin client screen's prior bitmap is to be shifted for reuse in ttie new 
screen. 

The reuse of a substantial portion of a screen display tiiat has been previously 
downloaded and drawn, made possible by ttie use of the scroll command, can substantially reduce 
tfie amount of data ttiat has to be downloaded to ttie ttiin client In scrolls whtoh Involved relatively 
small changes in position. This can substantially speedup tiie rate at which scrolled screens can 
be displayed on ttie ttiin client, particular In situations in which ttiere is a limited bandwidfli between 
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the browser and the thin client such as if they're communicating over the relatively slow digital 
cellular link common at the time this application is being filed. 

If the moved view window that results fbm a scroll or move command includes a portion of 
the web page's layout not currently In the virtual screen 10206, shown schematically In FIG. 102, 
function 10530 of FIG. 105A causes functions 10532 and 10534 to be performed. Function 10532 
scrolls the browser's virtual screen so that all of the view window will be contained within it, and 
then function 10534 requests a redraw from the browser for the newly moved virtual screen. Once 
this redraw is complete functions 10505 and 10510 will capture the newly drawn elements and will 
draw them, as has been described above with regaixl to FIGS. 108A through 106B. 

If the browser's proxy receives a zoom command from the thin client, ftjnction 10536 of 
FIG. 105A causes functions 10538 through 10552 to be perfonned. 

Function 10538 changes the display scale factor according to the zoom change. 

Function 10540 scales the view window relative to the browser's virtual window according 
to the selected zoom. 

Function 10542 checks to see if the scaled view window includes portions of the web 
page's layout not cun'ently contained within the virtual screen. If not it causes function 10544 to 
scroll the virtual screen or change its resolution to make the scale view window fit within the virtual 
screen. 

if scrolling the virtual screen will enable to new >rfew window to fit within the virtual screen, 
there is no need to re-layout the web page, and the zoom can be used to display the same layout 
as existed before the zoom, by showing a different location within it and/or by displaying it at a 
different scale factor. If, however, the zoom is a zoom out that causes the view window to be 
larger than Oie virtual screen size, in the embodiment shown in FIG. 105A, this will require that the 
web page be laid out at a new virtual screen size that allows the view window to fit entirely within it 
the virtual screen, so that the proxy browser can handle any input supplied to any portion of the 
view window displayed on the client as if it had occurred at a corresponding location on the proxy 
browser's virtual screen. In the embodiment being described, this may cause the web page to be 
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displayed at a new layout if the new virtual screen resolution is larger than the layout resolution 
used In the previous layout, and this can cause line breal<s to occur in different locations. 

In other embodiments of the invention, such as ones in which proxy browser was 
designed, rather than patched, to support zoomed views, and such as the ones described with 
regard to FIG. 115 in which the client zooms directly relative to a download of an entire layout, 
extreme zoom outs need not require a re-layout of the web page. 

Finally function 10552 calls for a screen redraw. This causes the screen capture and 
download routine to capture the redraw of the current view window with the new zoom scale footer, 
and download conesponding display information to the thin client so they can display the web page 
at the new zoom setting. 

As Indicated in FIG. 105B, if the browser's proxy receives a virtual resolution change 
command from the thin client, function 10554 causes functions 10556 through 10560 to be 
perfonrted. Function 10556 changes the browser's virtual screen resolution to the requested 
resolution. Then step 10560 calls for a screen redraw. This is because the browser rs-lays out the 
cunent web page at the new virtual screen resolution, and redraws ail of the cunent view window 
to be captured at the display's scale factor conesponding to the ratio between the number of pixels 
the view window has in the virtual screen relative to the number of pixels it has on the thin client 
screen. 

Such a change in virtual resolution changes the size at which a layout Is perfomned relative 
to the size of Images and text within such a layout Such a change In relative layout size changes 
the size at which Images and text will be displayed on ttie screen, unless ttie user maices a change 
in tiie relative size of ttie view window relative to ttie virtual screen tiiat cancels such a change In 
size. In tiie absence of such a compensating change in relative view window size, decreasing tiie 
virtual resolution Increases flie size at which Images and text will be shown on ttie screen, and 
tends to make tiie text lines shorter (Blative to Oie size of tiie fonts shown on ttiem, so as to allow 
more of text lines to fit on ttie screen at one time at a larger text size. Thus, changes in virtual 
layout size can be used to provide a certain type of zoom capability to tiie display of web pages. 

The inventors have found that quite good readability can be supplied using virtual screen 
of 640 by 480 when displaying web pages on a typical PDA-sized 320 by 240 screen, which 
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envolves scaling down the layout by a factor of 2. However, the Invention can be used to display 
web pages at even more reduced scales, such as displaying an 800 by 600 virtual screen 
resolution on a PDA-sized 320 by 240 display, even though readability will suffer, so as to enable a 
user to see how the web page might look when laid out for larger resolution displays. Of course, if 
the reader chooses to have the minimum font size limited, as was described above with regard to 
function 10614 of FIG. 106A, the text, even with such a large virtual resolution would still be shown 
with readable fonts, although the layout of the page would be quite different than that originally 
Intended for display at such a resolution, because of the relative Increase in font size that would 
result 

As Indicated by function 10562 of FIG. 105B, if the browser's proxy code receives other 
user Input from the thin client associated with a click on the thin client's screen, function 10564 
transfonfns the thin client screen position associated with the elide to the conesponding position on 
the virtual screen, and function 10566 relays the event to the browser's event queue so that it can 
respond to it as if the user had actually clicked on the virtual screen whldi most of the browser's 
code thinks it is laying out 

This is the method by which fte biDv^er on the proxy responds to input the user of the thin 
client makes to select most links, whether they be text links or Image links, on the web page 
displayed on the thin client For example, if the user clicks on a link displayed on the thin clients 
screen, the corresponding click will be relayed to the browser on the proxy, which will act as if the 
user had clicked on the same link in the virtual screen which it thinks It is displaying. The proxy's 
browser then responds by issuing an HTTP request over the Internet conesponding to the link. 
When the web page con-esponding to that link is received, the browser will lay out and seek to 
display it on the virtual screen, causing functions 10506 and 10510 of FIG. 105A to capture the 
infonnation contained in that the portton of the layout corresponding to the view window and to 
downtoad it to the thin client for display on it's screen. As a result, the user of tiie thin dient is able 
to surf the Web, In much the same manner as a user of a nonnal browsing computer. 

FIGS. 109A through i09C are highly simplified pseudocode representetions of code 10900 
on the thin dient computer designed to help it operate in conjunction with the proxy browser to 
enable ite users to browse the World WMe Web using its screen. 
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Function 10902 of FIG. 109A responds to the receipt of all or an initial portion of the 
downtoad stream sent to the thin client by function 10708 of FIG. 107. it does so by starting to 
respond to the indWIdual commands, of the type illustrated in FIG. 108. contained in that stream in 
the order in which they are received. It starts doing this as soon as one or more such commands 
are received so that the woric of drawing the new screen need not be delayed until the download 
stream has been fully received. The response to each different type of command contained In the 
download stream is indicated by the functions numbered 10904 through 10956 in FIGS 109A 
through 109B. 

As Indicated by functions 10904 and 10906, when the thin client reads a clear comniand In 
the download stream it causes the bitmap displayed on its screen to be cleared, or set to a totally 
white value. 

When the thin client reads a scroll command in the download stream, function 10908 
causes functions 10910 and 10912 to be perfomied Function 10910 copies the portion of the thin 
clients screen's bitmap that is to be reused after the scroll specified in the scroll command to a new 
position on that screen indicated by the XY shift value included in the command. Then function 
10912 clears the remaining portion of the screen. 

When the thin dient reads a bacl^ground color command in the download stream, functions 
10914 and 10916 set the current rectangle badcground color variable to the cotor specified in the 
command. This causes all rectangles drawn by the thin client in response to rectangle commands 
until the baclcground color value is changed to have that specified color value. 

When the thin client reads a rectangle command in the download stream, functions 10918 
and 10920 draw a rectangle, using the current background color, having a screen position, width, 
and height specified in the command. 

When the thin client reads an image locations command, functions 10922 and 10923 do 
nothing at that time. This Is because the bitmap's necessary to draw the image referenced in such 
an image locations command usually will not have been received at such time. In other 
embodiments, the browser associates rectangle drew commands with images, which will cause the 
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portion of the thin brovi^r screen associated with images to have a rectangle drawn on them 
Indicating where a bitmap Image is to t)e displayed. 

When the thin client reads a font command, functions 1 0924 and 10926 set the value of ail 
font attributes listed in the font command to the values listed for those attributes in that conrvnand. 
in different embodiments of the Invention different font attributes can be used. It is preferred that 
at least font family, font size, and font foreground color be supported font attributes. 

When the thin client reads a string command in the download stream, function 10928 
causes functions 10930 through 10940 to be performed. 

Functions 10930 tests to see if the thin client has In its font bitmap cache a bitmap for each 
character of the cument string in the cun:ent size and font family specified by the cunrent font 
attribute values. If not, functions 10932 thn)ugh 10936 are perfonned. 

Function 10932 sends an HTTP request over the thin clienf s Internet connection to the 
font server 134 described above with regard to FIG. 2. When the requested font is received from 
the font server, funcfions 10934 and 10936 place it in the thin clienf s font bitmap cash. 

It should be noted that some embodiments of the invention pemnanentiy store, as part of 
the thin client browser software, a sufficient set of font bitmaps so that the use of the functions 
10930 through 10936 are not necessary. In other embodiments, subpixel-optimized font outlines 
are either stored pemnanently by the thin client or are requested as needed, as are the font 
bitmaps in the example described In FIG. 109A. 

When the thin client has all of the font bitmaps necessary to render the current string, 
functions 10938 and 10940 draw the string using the current font attribute values including 
foreground color, upon the screen at the specified screen position. In the current embodiment font 
bitmaps are represented as alpha value bitmaps of the type described above v\rith regard to FIGS. 
60, 96, and 97. When doing so, the background color is driving the portion of the bitmap cun^ntly 
upon the screen over which the string is to be drawn. 
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In some embodiments, In order to reduce computation, the color value of the portion of the 
screen over which the string Is to be drawn is sampled at a relatively few number of points, and the 
average of those sampled color values Is used as the baclcground color for the entire string display, 
as is described above with regard to FIG. 97. 

In the embodiment being described, all of the strings contained in the download stream are 
single line text strings, many of which may have resulted from the wrapping of continuous text 
across line boundaries by the proxy browser's layout engine. As a result, in this embodiment, the 
thin client does not have to perform any such wrapping of text. 

Function 10940 draws a bitmap image of a strings by composing it from a plurality of 
separate font bitmaps corresponding to the letters of the string. Nomially In such composition each 
different separate character will be represented by a different separate font bitmap. 

It is preferred that the fonte used In such composition at different font sizes (such as 
different font sizes caused by changes In scale factor) have the shape and pixel alignment of each 
charecter selected to Improve readability at each such font size. In most embodimente this 
improved readability Is produced by selecting the character shape and position relative to a font 
bitmap so as to increase the alignment of the character shape with the pbceiation of the bitmap. 
Such shape and pixel alignment is particuiariy critical when dealing with font bitmaps of ten pbcels 
per em or less, and is even more criUoai at eight pixels per em or less. This is because as font 
bitmaps became smaller they become more difficult to read because of their more couree 
pixelation, and thus it becomes even more critical that they have character shapes and alignmente 
selected to fit such pbceiation. 

The fonte in these figures have a pixel size of eight pixel per em. A majority of the lower 
case lettere in this font fit within an advance width of four pixel columns of less. This width of four 
pixel columns or tess Includes the spacing, if any, that occurs between the shapes of charactere 
having such wWths. In these parflcular fonte, over eighty-percent the tower case characters of the 
roman alphabet fit within such an advance width These characters have an x-helght of more than 
four pixel row, which makes them generdiy considerably telter than they are vrfde. As a generally 
mte, such a relatively nanoow font can represent a larger amount of text within a given area at a 
given level of readability than a wider font. 
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In many embodiments of the invention ttie font bitmaps used by step 10940 at smaller 
scales are subpixel optimized bitmaps created by non-linear color balancing of the type described 
above, in which only color imbalances that occur within a pixel are distributed. When such 
subpbcel^ptimization is combined with shapes that have been properly shaped and aligned to 
better match their bitmap pixeiation, the resulting bitmaps drawn are amazing easy to read 
considering their pixel size. 

Returning now to FIG. 109B, when the thin client reads a control command from the 
download stream, function 10942 causes functions 10944 ttirough 10948 to be performed. 

Function 10944 tests to see if it already created a data or program object corresponding to 
the control ID specified in the current control command. If not function 10946 creates such a data 
or program object of the type specified in the control command and associates it with the control ID 
specified in that command. 

Then step 10948 draws a subpixel-optimtzed bitmap of the specified type of control object 
on the thin clients screen at the location specified in the control command. It then draws the text 
assodated with the control on the control objects bitmap using subpixel-optimized fonts. Then it 
associates a hot zone, having a display screen position conesponding to the control's bitmap, with 
the data object or program object representing the control on the thin client. 

When the thin client reads an image command from the download stream, function 10950 
causes functions 10952 through 10956 to be perfomied. 

Function 10952 scans the cunent display stream for all occunences of an image location 
command that has the same image ID as the cunent image command. For each such image 
location command, it causes function 10954 to draw the bitmap at the location specified by that 
image location command upon the thin clients screen. As with all the thin clients draw functions, 
any portion of the image which does not fit on the thin client screen is clipped in such draw 
operations. 
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Next func^on 10956 reims all other Items In the display list that occur at the same 
location as any of these drawn bitmaps. This is necessary because it is common for web pages to 
place text on top of images, and, thus, It te desirable that any strings which are Intended to be 
displayed at the same location as a bitmap image be redrawn after those Images are drawn, in 
one embodiment of the invention, the thin client merely redraws all non-image elements of the 
download stream's display list which occurs after the first image location command in that ilsL 

If the user clicks on a hot zone 11000 associated with a text entry field, as indicated In 
FIG. 110, functions 10958 and 10960 of FIG. 109B cause a keyboard routine comprised of 
functions 10962 through 10978 to be executed. 

Function 10962 displays a pop-up user keyboard 11102 and text edit field 11104, 
illustrated in FIG. 111, on the thin clients screen. Then a loop 10964 is perfonned until the user 
presses the enter key on the pop^up keyboard. During this loop each time a user types a text 
character, function 10966 causes function 10968 to place a subpbcel-optimlzed text bitmap of the 
character on the pop-up keyboard's text edit line at the cunent cursor posltfon and moves the 
bitmap of the cursor to a position after the newly drawn character, and then function 10970 adds 
the typed character to a temporary text edit string associated with the popnip keyboard's 
programming. 

When the user presses the enter key of the* pop-up keyboard, function 10972 causes 
functions 10974 tiirough 10978 to be performed. Function 10974 stores the value of the temporary 
text edit string associated with the pop-up keyboard in the text edit control for whteh the pop-up 
keyboard has been evoked. Then function 10976 draws the characters of that text edit string, 
using subplxei optimized bitmaps, in the bitmap of the text entry field 1 1000 of the control object on 
the thin client's screen, as shown in FIG. 112. 

Then function 10978 removes the pop-up keyboanJ fiDm the thin clients screen by 
drawing over it the bitmap which was displayed on-screen before the pop-up keyboard was drawn. 

FIG. 113 illustrates that the pop-up keyboard routine can be used for other purposes 
besides entering text in text entry field. Although it is not represented in the pseudocode of FIGS. 
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109A through 109C, the pop-up keyboard can also be used to enter the URLs of web pages a user 
would like to see displayed on the thin dient 

FIG. 114 is virtually identical to FIG. 113, except it illustrates an embodiment of the 
invention which has a button bar, or Toolbar, at the top of its graphical user interface. This button 
bar includes at its leftmost end back and fonvard buttons of the type commonly found in Web 
browsers. It also includes buttons labeled R B, and H, whidi correspond to a refresh button, a 
bookmark button, and a history button, which are also functions commonly found on Web 
browsers. The button bar also Includes an URL text entry field, which if clicked will cause the pop- 
up keyboard shown In FIG. 114 to appear. When the pop-up keyboard is not being displayed, this 
text entry field displays the URL of the cunrent web page displayed on the thin dienf s screen. In 
one embodiment of the Invention a user can select whether or not to display such a toolbar by 
pressing a hardware button. In this embodiment even when such a tool bar is not shown the user 
can use hardware buttons to invoke some of the mre common web browsing fundions, such as 
the back command and fonvard commands. 

In other embodiments of the invention, sudi a graphical user interface Toolbar would 
preferably also Include buttons or menus allowing the user to access other functionality of the 
browser, Including changing the zoom and/or relative layout size of a web page's display. 

Retuming now to FIG. lOSB, If the user clicks on a hot zone of a button or menu item 
control, funcHon 10980 causes functions 10981 and 10982 to be perfonned. 

Function 10981 changes the appearance of the button or menu item appropriately. In the 
case of a button, the bitmap associated with the button is redrawn to Indicate the button is being 
pressed. In the case of a menu item, either a submenu vAW be display, or the display of the menu 
item will be removed, depending upon whether or not a final selection has been made. 

If a final selection has been made in the case of a menu item, or the button has been 
pressed and released, function 10982 sends the button's or menu item's control ID and an 
indication that it has been selected up to the browser, which responds by causing the 
conesponding button or menu item control object on the biowser to act as if it had been clicked. 

If the user clicks on the hot zone associated with another type of thin cHent control, 
function 10983 changes the appearance of the control's bitmap on the thin cGenfs display 
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accordingly. For example, In the case of a check box, a check would either be displayed or 
removed from the display of the control on screen. Then step 10985 stores the conesponding 
state change in assodation with the control object. As stated above, in the embodiment being 
described, the state of such control objects are not communicated to the browser until the browser 
requests such Information, in order to reduce communication demands. 

If the user clicks on any other portion of the thin client's screen not associated with the 
control interface of the thin client program or its computer, functions 10986 and 10987 send an 
event conesponding to that dick up to the proxy browser atong with the screen location at which it 
occurred. As was described above with regard functtons 10562 through 10556 of FIG. 105B, the 
browser vwll transform the location of such a dick to the corresponding k}cation on its virtual 
screen, and will respond to such a dick as if rt occuned upon the screen which the browser thinks it 
is drawing at the resolution of the virtual screen. In some embodiments, to further reduced 
communication demands, the thin client vAW only report such other dicks to ttie browser if it has 
reason to believe they conesponds to a user input the prox/s browser is supposed to respond to. 

Referring now to FIG. 109C, If the thin client receives a query from the proxy browser 
asking the state of one or more control objects, function 10988 causes fiinctton 10989 to query the 
state of the corresponding controls on the thin dient, and function 10909 to transmit that state 
information to the browser. As was described above with regard to functton 10518 of FIG. 105A, 
the proxy browser will then retom such requested information to ttie part of flie browser which 
requested it, as If that infbnnation were part of the current stete of tiie conesponding control - 
objects associated witti the virtual screen. 

if Uie user of ttie ttiin client enters the command to scroll its screen, functions 10991 and 
10992 upload that scroll command to Uie proxy. This causes the functions 10520 ttirough 10534 
described above with regard to FIG. 105A to generate and downtoad a new download stream for 
ttie display of ttie cunent web page at tiie newly scrolled position. 

If the user enters tiie command to change ttie zoom, i.e., scale, of the image displayed on 
tiie ttiin dient, functions 10993 and 10994 upload a corresponding zoom command to ttie proxy. 
This causes ttie functions 10536 and 10552 described above witii regard FIG. 105A to cause a 
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new download stream to be downloaded to the thin client for display of the cunrent web page at the 
new zoom setting. 

If the user enters the command to change the ^rirtual resolution of the thin dienrs display, 
that is to change the resolution at which the virtual screen on the proxy browser is laid out, 
functions 10995 and 10996 upload the selected virtual resolution to the proxy. This causes 
functions 10554 through 10560. described above with regard FIG. 105B. to have the virtual screen 
re-laid out at the new resolution and a oonBsponding downloads stream to be sent to the thin 
client, so it can display the portion of ttie virtual sopeen oonBsponding to the window at the current 
zoom setting upon the thin clients screen. 

As indicated at the bottom of FIG. 109C, if the user enters another command associated 
with the thin ciienfs control graphical user interface, function 10997 vdil cause a conespondingly 
appropriate response, indicated by the ellipses 10999 to be perfbnned. Such other functions can 
include the selection of bookmarks, the accessing of bookmariced web pages, back and fonA/ard 
functions, or any other function whid) can be part of a browser's user interface. Such demands 
can be selected by use of physical buttons or other physical inputs on the thin client computer, by 
the selection of graphical objects, such as buttons, menu item, or dialog box controls, or virtually 
any other known graphical user internee technique. 

Figures 115 through 118 relate to an alternate method for enabling a thin client computer 
to browse the web through a proxy sender. In this embodiment the entire layout 10206 of a web 
page created by the proxy computer is downloaded to, and cached by, the thin client, as indicated 
in FIG. 117. As will be described, this allows the thin client to scroll within the layout at 
substantially higher speeds, although it can increase the total number of bits downloaded, since it 
attempts to download the entire layout and all images for each web page viewed. 

FIG. 115 illustrates portions of the proxy browser's code 11500 tiiat can be used wiUi such 
a page layout caching scheme. 

In this embodiment of ttie invention if ttie proxy browser receives a request for a web page 
from tiie ttiin client, functions 1 1502 causes functions 11504 ttirough 1 1524 to be peri'onned. 
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As indicated by the pseudocode associated witii function 11502 in FIG. 115. in this 
particular web caching embodiment the thin client can request a web page with a desired view 
setting for that page, Including a desired virtual resolutioni zoom setting, and view window position. 
This is done to allow a user to associate such view settings with a boolonarlc, including a particular 
URLs or a portions of a URL path name, so as to allow the user to automatically see such web 
pages at a desired virtual resolution, zoom setting, and view window position, without having to 
separately enter such setting values each time ttie page Is requested. This, for example, would 
allow a user view commonly accessed web pages witti tiie display automatically zoomed in on a 
desired portion of ttiat page using text of a desired size. 

Once a request for a web page has been received from ttie ttiln citent, function 11504 on 
tiie proxy browser requeste tiiat web page from tiie server identified in tiie URL of ttie request from 
ttie ttiin client When ttie web page is received flrom the server function 11506 causes functions 
11507 ttirough 11516 to be perfomied. 

Function 11507 causes tiie layout engine of tiie browser on ttie proxy to lay out ttie 
received web page at ttie virtual screen resolution associated witti ttie view setting specified in ttie 
web pages request This layout is made using scaled string measuremente for substttuted fonte, in 
a manner similar to ttiat described above witti regard to functions 10606 ttirough 10618 of FIG. 
106A. The scale factor used is determined by tiie view setting specified in request for ttie cunent 
page. 

Function 11508 specifies a virtual screen position relattve to ttie resulting layout which will 
indudettie view window implicit in ttie view setting of ttie current request Thus for example, ifttie 
view setting requeste to see ttie rightmost portion of a 640 by 480 virtual screen resolution output, 
and ttie layout is forced to have a widtti of 800 pixels, ttie virtual screen position would extend from 
approximately pixel column 160 over to pixel column 800 in ttie layout 

Function 11518 causes functions 11520 to scale and subpixel-^ptimlze each image 11702, 
iliusbBted schematically In FIG. 117, received in association witti ttie web page being laid out 

Once all tiie images referenced in tiie web page have been received, scaled, and subpixel 
optimized, function 11522 causes function 11523 to create a display list for ttiat layout, and to 
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compress that display fist and all its associated subpixel-optirrdzed, scaled-down images. Then 
function 1 1524 transmits that compressed data to the thin client in a download stream that includes 
the web pages layout, followed by its scaled-down, subpixekptimized images. 

If the user receives a request from a thin client to rescale and subpixel-optlnnized one or 
more images previously downloaded at a different scale, function 11526 through 11532, rescale 
and subpbcekptimize, compress, and download such images to the thin client. This allows the 
user to view the web page at a different subpixel optimized size if he or she seeks to view the 
downloaded web page layout at a different zoom setting. 

If a screen Inputs event is received from the thin di'ent function 11534 causes functions 
11536 through 11542 to be perfomfied. 

Function 11536 tests to see if the page layout coordinate associated with the command 
con-esponds to a portion 10206A of the web page layout 10206 currently mapped to the proxy 
browser's virtual screen 10208, shown In FIG. 117. If not, function 11538 sculls the virtual screen 
to a new portion 10206B of the. layout that Includes the layout coordinates associated with the 
command. 

Function 11540 calculates the virtual screen coordinate conesponding to the page layout 
coordinate of the received screen event. Then function 11542 places the Input screen event with 
its virtual screen coordinates in the browser's event queue, so that it can respond to that event, 
such as the clicking of a link, as if the user had clicked at Ifs the conesponding virtual screen 
coordinate on the virtual screen Itself. 

FIG. 116 is a highly simplified pseudo code description of portions of the thin client code 
that can be used to support the page layout caching scheme illustrated in FIGS. 1 15 and 1 17. 

If the thin client starts to receive a download stream containing a page layout's display list, 
function 1 1602 causes function 1 1604 and 1 1606 to be perfonmed. 



wo 02/088979 



PCTAJS02/14380 



-83- 

Function 1 1604 sets the mapping of the view window (such as the view window 10210A 
shown In FIG. 117) relative to the page layout, and then calculates the cument display scale factor 
based on that niapping. 

Function 11620 displays any portion of the downloaded page layout that falls within the 
current view window, using the current scale factor. This process includes the functions 11622 
through 11630. 

Function 11622 displays each string elenrent that occurs within the current view window 
with font sizes that are a function of the cunent scale factor. When it does so it adjusts, if 
necessary, for any disproportionate changes in the relative size of characters that might result 
from the uneven effects of font hinting as the pixel size at which such characters are displayed 
changes, it does this by using techniques for compensating for the discontinuities and 
disproportionaiities, such as changing spacing between characters, similar to thosetraditionally 
used to provide a WYSIWYG conrespondence between the display of text on a computer screen 
and its appearance when printed at a much higher resolution, if font bitmaps having a different 
size that than previously displayed are required, font bitmaps for such differently sized characters 
can be either accessed from storage on the thin client, accessed from a network font server, or 
rasterized at the needed size from a font outline. 

If the display created by funcfion 1 1620 is at a different scale factor than that for which the 
bitmap images 10818 have been scaled-down, function 11624 causes functions 11626 through 
11630 to be performed. These request the proxy server to rcscale at the new scale factor and 
subpixel-optimize all images that are totally or partially within the view window. Then the bitmaps 
of the same images are locally rescaled from the fonfnerly downscaled and subpixel-optimized 
images 10818 stored on the thin client and displayed on the thin client screen to provide a 
temporary representation for such images. Then, when the requested images which have been 
rescaled from flie original, higher resolution bitmaps associated with the web page have been 
received by the thin client from the proxy server, they are drawn at the appropriate location on the 
display screen. 
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In some embodiments, when a user changes the zoom of the display, the bitmaps of any 
images corresponding to a portion of the page on the screen at the new scale factor are displayed 
with a quick, but crude representation of the image generated on the thin client to provide the user 
a temporary representation of sudi images to be used until the property subpixel-optlmized 
versions of the images have been downloaded. Such quick representations are relatively simple to 
generate when the new scale factor Is an integral ratio of the sc^ie fiactor of the previously 
downloaded bitmaps. When this is not the case, the temporary representation could be produced 
in any of a number of ways. These include displaying them as Images having integral scaling ratio 
tliat have been either cropped or scaled to an integral ratio smaller than the proper scale so to not 
take up more space than the properly scaled images that are intended to over draw them. 

^ If the user generates a screen Input to be sent to the proxy browser, function 11632 
through 11636 transform the thin client screen coordinate of the input to a corresponding page 
layout coordinate. Then the screen Input and corresponding page layout coordinate are uploaded 
to the proxy browser. The proxy browser then responds to such an Input using the functions 11534 
through 11542 described above with regard FIG. 115. This causes the proxy browser to respond 
to such screen input as if the user had clicked on a conBsponding portion of the web page on the 
proxy browser's virtual screen. 

Caching schemes, such as that just described with regard FIGS. 115 through 117, that 
allow the thin client to store more than the portion of a web page currently displayed on a screen, 
can be used to allow a user to scroll and/or zoom more rapidly relative to web page's content This 
Is particularty true if the thin client has a relatively low bandwidth to its proxy server. 

The embodiment of such a caching scheme which has just been described operates 
relatively well even with bandvrtdths as low as those associated with current digital cellular 
communication rates commonly available at the time of this application. This is because all the 
content except images, including In most web pages can nonnally be compressed to fit into 3,000 
bytes or less. Thus, at such cunrent digital cellular communication rates, such the entire text 
portion of most web pages could be downloaded in several seconds, and the initial portions of it 
could be drawn in even less time. Of course the downloading of the images might take more time. 
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but ail but laige Images at ttie start of the web page would commonly be displayed within a few 
seconds. And with foster communication links this delay can be reduced tremendously. 

FIGS. 118 through 120 fllustrate aspeds of the Invention that can be used in virtually any 
Web browsing environment, but which are paitlculaily useful when browsbig the Web on small 
screens. This includes use on small screen devices such as the thin client computers discussed 
above. Because these aspects of the invention invoh/e a zoom into or out of a selected portion of 
a web page, they can be made to work quickly on such thin client computers by use of a layout 
caching scheme of the type just described. 

FIG. 118 shows the view of a standard web page that has been laM out at a virtual 
resolutton of 640 by 480 and then downscaled and 8ubpixel^)pt]mlzed for display on a 320 by 240 
screen. Such content Is readable by those with good eyes at the distance at which most people 
commonly use handhekl computers. However the content of most Web pages can be made even 
easier to read If It is displayed at a larger sb». Since most Web content is laid out In a pluraliV of 
columns, it would often be desirable to be abte to quickly zoom a display to the top of a column at 
whkih a user wouM like to stert reading. In the interface shown In FIG. 119 and 120 a user can do 
this by dragging the polnthg device 11902 across the desired text column at a verttoal positton 
whk^ the user would like displayed near the top of the display screen In the zoomed view. When 
the display is in the mode to perfonn this type of zoom, a horlzontel linear drag of the type shown 
In FIG. 1 1 9 will cause the display to scale the width of the web page layout indicated by the drag to 
fit the wMth of the screen. In the example shown in FIG. 119 this user input wouM cause the 
display to be zoomed as shown in FIG. 120. 

Preferably the user interface also allows a user to drag a selection box around an area in 
tiie web page layout shown on the screen, and the system will zoom the display of the web page 
so that the selected area in the wsb page fife the screen. 

It is also prefened that In such drags the user be allowed to drag the pointing device 
across a boundary associated with each edge of the screen, and if this is done the portkm of the 
web page shown on the screen will scroll in response, to allow the user to select to perfonn a zoom 
to fit to a width, height, or area within the web page layout that Is too large or improperiy posittoned 
to fit totally within the screen as the stert of such a drag. If such a drag selecte a portton of the 
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layout too large to fit on the screen at the scale fector displayed during the drag, It would change 
the scale factor so as to decrease the size at which text and Images were displayed. 

When we say that a selected width, height, or area in the layout is scaled to tit the screen, 
we mean that tt is scaled to have ite largest dimension ranges between two thirds and the full 
conesponding dimensions of the screen. Nomially It would be prefemsd that sudi scaling make 
the selected length or area have a largest dimension that ranges from eighty or ninety percent to 
the fiill conesponding dimension of the screen. 

FIGS. 121 to 128 illustrates a feature of the invention called zoom-click. This feature 
aOows a user to more easily and accurately select items within a screen which is seen with a low 
resolution, which is very small, or which is being used with a pointing device that can not be as 
easily positioned with accuracy relative to desired locations on a screen. This is partteularty useful 
with dealing with cell-phone sized sheens, with touch screen devtees using fingers as a pointing 
device, and for touch screen devices used in an environment such as a moving car in which it is 
difficult to accurately place the pointing device. 

In zoom click when the user clicks down at a given location in a screen, the portion of the 
screen upon v\rtiich he or she has clicked is shown at an expanded scale. The user is then free to 
navigate in this expanded representation with the pointing device held down until the pointing 
device is in the desired location. At this point the user can stop pressing down, and release the 
pointing device, causing the cument location at the time of the release to be treated as the selected 
location for purposes con^esponding to a traditional graphical user interface click. 

With zoom click double clicks can be represented in different ways. One of the easiest is 
merely to record a quick secondary click and release shortly following a zoom dick and near the 
same location as the zoom dick as converting the zoom click Into a double-click. 

Although other embodiments of zoom dick it can operate differently, the prefened 
embodiment causes pointer movement In the enlarged view during the down dick In a zoom dick 
to occur at the same rate as nonnal pointer navigation. This means that a user will have 
approximately twice the pointing resolution as he othenvise would. It is also prefened that when a 
user gets to the edge of the screen while moving the pointer during a sustained down dick in zoom 
dick mode the Image will scroll to enable the user to navigate the entire page in this mode. 
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In the example of FIGS. 121 through 128, a clamshell cellphones/computer 120C is 
shown. In this example, It is assumed that the cellphones has a 320 by 240 whole pixel resolution 
and color subptxel addressability. Of course in other embodiments of the invention other 
resolutions could be used. For ease of use, the cellphone Is assumed to have a touch sensitive 
screen that can be operated by users finger. 

FIG. 121 shows the cellphones with the same image of the priceline.com Web page 
shown in FIGS. 11 and 110. 

FIG. 122 shows vtdiat happens when, while in zoonHdick mode, the user tries to press his 
or her finger 12102 down upon the screen to select the same text entry field 11000 discussed 
above with regard FIG. 110. It is often difficult to estimate in advance the exact locafion at which a 
cursor will be placed when one touches one's finger to a touchscreen. Zoom click helps with this, 
since it altows the user to see the position of the cursor 170 which results from a finger touching 
the touch screen before any selection is made. It also shows the screen at a larger view scale to 
make it easier for the user to position the cursor 12204 on the desired link or control, such the 
desired text entry field 11000 shown in FiG. 123. Once the user renK)ves his finger from the 
touchscreen, as shown in FIG. 124, the pop-up keyboard 11102 appears, Just as it did In FIG. Ill, 
described above. 

As shown In FIG. 125, when the user, while in zoom click mode, tries to touch a desired 
letter In the pop-up keyboard 11102, in this case the letter "b," the image of the portion of the 
keyboard touched increases its scale. In the example shown in FIG. 125 the user has not exactly 
position the cursor 12204 at the desired locatton at the time of his initial pressing of the 
touchscreen. The user can easily conect this problem in zoom-click mode by dragging his finger 
12202 to position the cursor 1 1102 at the tocatlon shown in FiG. 126. If the user then removes his 
finger from the screen, the screens image will revert to Its nonfnal scale, which enables the entire 
pop-up keyboard 1 11 02 to be shown to allow rapid selection of the next character, no matter where 
it might lie within the keyboard. 

As shown In FIG. 127. the letter "b" selected in FIG. 126 is shown as having been entered 
in the pop-up keyboard's text entry field 1 1 1 04. 
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If fhe user continues selecting characters by the process shown in FIGS. 125 through 127 
he or she will be able to enter an entire string of text followed by the selection of the enter key 
whidi will cause the desired text to be entered at tfie desired location in the web page, as indicated 
by FIG. 128. 

In many embodiments of the invention, the zooms used in zoom click involve expanding 
the bitmap previously shown on all or part of the screen by an Integral ratio, such as 2x or 3x. This 
allows such zooming to be perfomfied virtually Instantaneously, even by relatively low powered 
processors, making zoom click a very rapid user interface. 

FIGS. 129 through 137 illustrate an aspect of the invention that enables a user of a Web 
browser to select a portion of text from a web page to be the reflowed, or re-laid out, across line 
boundaries at a substantially larger scale factor. Such reflowing of text is particulariy useful on 
displays having small screens, since it allows selected Web text to be displayed with much laiger 
fonts, while at the same time allowing entire lines of such text to fit within such screens. This 
enables such lines to be read quickly, without the need to repeatedly horizontally scroll back and 
forth to read successive lines of such text. 

Regardless of how high the resolution of a small screen display is, the human eye can only 
see what it displays at a relatively large resolution if the display is held relatively close. This aspect 
of the invention enables Web text to be display wrapped across lines that fit within the width of a 
display at a relatively large scale factor. For example, it allows the user of a handheld computer 
with a four inch diagonal screen to display text at a sufficiently large scale to be seen by a group of 
people who are standing five or six feet away. Similariy it would allow a user to view text on 
cellphone or wristwatch sized display without having to hold them close to his or her face. It can 
also be used v^th nomrial sized computer display screens to display Web text to people who are at 
a relatively large distance from the screen, or who are visually impaired. 

FIG. 129 provides a highly simplified pseudo code description of programming 12900 
which could be used by a client computer to redisplay Web text according to this aspect of the 
invention. 
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It should be understood that this aspect of the Invention is not limited to use on client 
computers. In fact, with modification this aspect of the Invention can be used In viewing visual 
output generated by applications other than Web browsers, such as in systems of the type 
described below with regard FIG. 140 and 141. 

Many web pages are designed to have their text laid out in different columns, that is, in 
different horizontal positions relath^e to such a layout A web page can indication such different 
desired horizontal displacements In multiple different ways, Including the use of tables and frames. 
It is preferred that the method shown a system used with the methodis capable of displaying a web 
pages text in such a multi-column layout which reflects such indications of different desired 
horto)ntal displacements. 

If the user selects an area of a web page layout for text reflow in a single column at a new 
scale factor, function 12902 causes function 12,904 through 12908 to be performed. The If such 
a selection is made when a web page's content is being displayed in a relatively nonnal manner 

In some embodiments of the invention such a selection is made by dragging a pointing 
device, such as the stylus 11,9002 shown in FIG. 130 across the portion of the width of a web 
page which the user desires to have reflown at a laiiger scale across the width of a display screen 
or window. This Is similar to what was discussed above wifli regard to FIG. 119, except that the 
method currently being discussed allows text to be re-flowed across line boundaries, letting a 
selected column of text be displayed with much laiiger fonts while at the same time aDowing whole 
lines of such text fit within the screen. 

Function 12904 of FIG. 129 selects ail strings and corresponding underlining (i.e., labeling 
of text as bing a link) in the layout of the cunent web page which are substantially within the 
selected layout area. 

FIG. 131 Illustrates the top portion of the layout 10206A, similar to that shown in the 
bottom half of FIG. 117, of the web page shown In FIG. 130. In FIG. 131 ttie dashed rectangle 
13,102 represent the portion of the web page's layout comesponding to the column selected by the 
user In FIG. 130. 
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In some embodiments of the invention, a string vriil be considered to be within the selected 
area only if a substantial portion, such as two-thirds or three-quarters of its length fits within the 
area selected by the user. For example, in FIG. 130, the user intended to select the text at the 
right hand portion of the screen displayed in that figure. However, In the example of FIG. 130 the 
user foiled to exactly select the width of that Intended column with the drag of the stylus. 
Nevertheless, because function 12904 selects all strings which are substantially wittiin tiie selected 
area, ttie text reflow will operate as If ttie user had selected exactly ttie intended column. 

FIG. 132 illustrates an initial portion of ttie strings in ttie layout of tiie web page shown in 
FIG. 131 ttiatfall wittiln ttie selected area. In ttiis figure, underiining indicates portions of text which 
conespond to linics. 

Once ail of ttie strings in tiie selected area have been selected, fundion 12906 labels any 
groups of one or more successive strings whose closeness in ttie layout or ottier characteristics 
indicate ttiey are part of flie same paragraph. This Is Indicated in FIG. 132 by ttie paragraph 
brackets 13202. 

As indicated In FIG. 132, tills mettiod might not detect all groupings of fonts which are 
paragraphs, but it does detect many of ttiem wittiout ttie need to reference ttie HTML 
con-esponding to the text In ttie embodiment of ttie invention being described, such HTML Is 
stored on tiie proxy server, meaning ttiat such an access would require ttie delay associated witti 
communication between tiie client computer and a proxy server. In ottier embodiments, 
partlculariy tiiose in which tiie client computer has a full browser resident upon it, or ttiose having a 
high access bandwldtti link to a proxy server, access to ttie HTML code could be used to more 
accurately determine how ttie selected strings should be grouped in paragraphs. In oflier 
embodiments, ttie layout infomiation downloaded to a proxy server could contain any information 
about paragraph boundaries contained wittiin a web pages IfTML. 

Once ttie selected strings have been grouped Into paragraphs, function 12908 reflows the 
text of each paragraph using ttie selected expanded scale factor across tiie vridtti of tfie display 
screen (or display window if ttie image is being shown on less ttian an entire display screen). 
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This text reflow process is illustrated In FIG. 133, in which the strings in the top portion of 
FIG. 132 are laid out at a new scale factor. In the example being shown, the text of FIG. 130 has 
been selected to be reflown at twice Its original size within the same screen. Preferably the user 
interface of the thin client allows the user to select a piurality of different scale factors for use with 
the selected-text-reflow function, ranging from rather modest to rather extreme Increases In font 
size. 

In FIG. 133 underlining is used to represent something different than it does in FIG. 132. 
in FIG. 133 the text on each line which came from a common layout string in FIG. 132 is shown 
with continuous underiinlng. Underlining gaps between portions of text on the same line in FIG. 
133 which come from different strings In FIGS, 132 are exaggerated to make their difference more 
readily visible. In FIG. 133 all of the Individual strings from the original layout shown In FIG. 132 
that have been wrapped across a line boundary are indicated by a arrow from their portion on one 
line to their following portion on the next line. 

FIG. 134 provides a schematic iliustretion how the selected strings of the original layout 
shown in FIGS. 130 and 131 look once ttiey have been reflown at approximate twice the size on 
the thin dienfs screen. As can be seen by looking at FIGS. 134, such a text reflow makes Web 
text much easier to view from a distance. Refiowing the same text at 4x or 6x instead of 2x would 
make it possible to show the same web content to people at quite a dtetance from the display 
screen. 

FIGS. 135 through 137 illustrate other methods that can aliow a user to select a portion of 
text to be reflown. 

FIG. 135 illustrates a portion of a web page having a central column of text which has 
intiuded Into that column one or more portions of o&ier text. 

FIG. 136 Illustrates how the user obtained a zoomed out view of the entire web page's 
layout. In many actual embodiments, text greeking woukl be used to Indicate portions of text too 
small to be represented In such zoonH)ut views as Indh^ldual charactere. Such a zoomed-out view 
could be generated quickly on thin client computers such as those described above with regard 
FIGS. 115 through 117 In which a web page's entire layout was stored on the thin client, itself. 
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In FIG. 136 the user has selected a mode whidi allows her or him to define a polygon 
shaped area upon the zoomed-out web page view by clicking the display of the web page at 
comers in such an area. Once this is been done, the selected area will be used by the function 
12904 shown in FIG. 129 to select which text Is to be reflown. 

FIG. 137 illustrates how the selected text will appear once it has been reflow and 
displayed. 



FIGS. 138 and 139 provides more description of the font server 230 described above with 
regard FIG. 2. 

FIG. 138 conesponds to FIG. 2. except that In it there are plurality of the client browsers 
200 each of which accesses content from one or more servers 220 through the same proxy server 
21 0 and each of which accesses fonts from the same font server 230. 

TTiis is because the software sold, licensed, or distributed for use in each of the thin client 
browsers has been programmed to seeic fonts such clients do not have from the same font server 
230 and to make Web requests through the same proxy server 210. Of course, in other 
embodiments of this aspect of the invention the thin clients couW be programmed to select which 
of a common plurality of proxy servers to use based on such factors as their geographic location, 
or their Internet service provider. S War considerations could be used by the thin clients to select 
from which of a common plurality of font servers they are to request and receive fonts. 

FIG. 139 provides a highly simplified pseudo code description of programming 13900 that 
can be used on a font server of the type illustrated in FIG. 138. This font server could also be used 
by nomial browser computers, as well as by computers mnning applications other than Web 
browsers. 

If the font server receives an HTTP request from a computer for a character of a particular 
font, funcfion 13902 causes steps 13904 through 13922 to be perfomied. 
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The particular embodiment of font server code shown In FIG. 139 Is designed for use with 
a protocol which specify each character desired for particular font at a particular size with a 
separate HTTP request. It specifies the desired font, font size, and character as part of a URL 
pathname. Of course, in other embodiments font servers could allow HTTP request to specify 
more than one font, and coufd specify fonts other than with URL pathnames. 

In systems which request each character-font shapes separately, it is preferable that the 
HTTP protocol 1 .1 or later be used, since it allows multiple HTTP request to be handled by a server 
from a given client computer without having to open and close a separate connection for the 
handling of each such request. 



In the embodiment of the invention shown in FIG. 139, if the font server determines that it 
currently has stored a font bitmap corresponding to the URL pathname specified in the request, 
function 13904 causes function 13906 to send that font In an HTTP response to the network 
address from which the URL request came, and then function 13908 charges an account 
associated with a transaction. Such a downloaded font could be either a font bitmap or a font 
outline description. 

Such a charging an account is not used in all embodiments of the invention. In some of 
those in which it is used, the account charged Is one associated with the computer to which the 
font is sent In other embodiments, the charge is to an account of a party associated with the web 
page which Included a specification for such fonts. In yet other embodiments, the charge Is to an 
account associated with a proxy server of the type described above, or to a user of the services of 
such a proxy server. 

if the requested font is not in the font server's storage and it Is a bitmap for which the font 
server has a corresponding outline font, fundion 13910 causes function 13,912 through 13,922 to 
be perfomned. 

Function 13912 generates a font bitmap having the attributes, such as size and possible 
transfomnation, Indicated by the font pathname of the HTTP request This function includes 
detennining if the requested fonf s pathname indicated that a subpbcel-optlmized version of the font 
Is desired. If so, function 13914 and 13916 generate a subpbcel-optlmized version of flie font, 
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preferabl/ using the non-linear color balancing method described above with regard FIGS. 55 
through 96. 

Once the font bitmap has been created, function 13918 sends the bitmap over the network 
in an HTTP response to the requesting address. Function 13920 caches the font bitmap at an 
address conesponding to the pathname specified in the request Function 13922 charges an 
account associated with the transaction, as discussed above with regard to function 13910. in 
embodiments where such charging is perfonmed. 

FIG. 140 fliustrate that certain aspects of present invention can be used to enable a thin 
client computer 200 to display digital content conesponding to the text and the images generated 
as screen output by one or more applications running on upon a remote computer 14000. Such 
applications can Include Web browsers, spreadsheet, word processors, database programs, or 
virtually any other type of software capable of generating screen display. 

The remote computer includes remote screen generator programming 14006, which 
includes hoolcs in the dispateh table 14008 of the remote computer's operating system 14004. 
These hooks intercept calls made by one or more of the applications 14002 to the operating 
system to draw text, shapes, lines, control objects, and bitmap's to a screen at a given display 
resolution. In some embodlmente, such draw commands will actually cause content to be 
displayed on a screen associated with a romote computer, in others there will be no screen at the 
remote computer, and thus such draw commands will be made to a virtual screen. In the text that 
follows, for purposes of sinr^licity, I wiil refer to the video space to which these application thinks 
they are displaying graphic output and receiving user input on a given client computer as a virtual 
screen. 

When one of the applications 14002 request the operating system to draw a display 
element, that call is intercepted by one of the hooks in the operating system's dispatch table, so as 
to make a comesponding call to a corresponding routine 14010 of the remote screen generator. In 
a manner similar to that described above with regard to FIGS. 102 and 106A through 106C, this 
causes a download display list 10212A to be created which is substantially similar to the display list 
10212 described above with regard to FIG. 102 and the figures that follow it. A zoom, scroll, and 
virtual layout control 1412, conesponding to the controls 10214 through 10218 shown in FIG. 102, 
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controls the mapping of the thin clients view window into the virtual screen and. thus, the display 
scale fector at which the elements drawn by an application Into the virtual screen are drawn and 
positioned in the download display last 10212A. Preferably this includes subpbcel optimization of 
image bitmaps, and font substitutions of the type described above with regard to FIGS. 106A 
through 106C. 

Once the download display list has been created for a given virtual screen, it is 
compressed and downloaded to the conresponding client computer, which then draws it upon its 
screen in much the manner described above with regard to FIGS. 109A through 1090. 

In some embodiments of the invention, indh/idual draws to the virtual screen will have 
conresponding draw commands downloaded to the thin client. This, can be used to speed the rate 
at which minor changes to the thin clients screen can be ntade in response to conresponding 
changes to the virtual screen. 

In the embodiment shown in FIG. 140. user input associated with screen bcatlons are 
uploaded to the remote computer from the thin client, and they have their screen coordinates 
transformed to reflect the mapping between the thin clients view window and virtual screen. Once 
this is done such events are placed In the event queue 14014 of the remote computers operating 
system with their transfomied screen coordinates so the associated application 14002 will respond 
to that event as If It had been entered upon the remote computer's corresponding virtual screen. 

Many of the techniques used by screen sharing applications, such as LapUnlc or 
pcAnywhere can be used In conjunction with an embodiment of the invention of the type shown In 
FIG. 140. In feet, when the remote computer in that figure has its own screen, the embodiment 
shown In FIG. 140 can be used to perfomi screen sharing bebAfeen the client computer and the 
remote computer. 

It should be appreciated that In embodiments in which the client computer has a 
reasonable amount of a computational power, the client and the remote computers can operate in 
a peer-to-peer manner. The remote computer can be a dedicated application server computer or it 
can be any other type of computer, such as a personal computer. Including a desktop computers, 
laptop computers, or tablet computers. 
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FIG. 141 illustrate an embodin^nt of ttie invention that Is somewhat simDar to shown In 
FIG. 140, in that it uses hooks Into the dispatch table 14008 of a computer's operating system 
14004 to intercept operating system calls made by one or more applications 14002 in order to 
cause the screen displays generated by such applications to be scaled-down and/or subplxel- 
optimized according to aspecte of the Invention described above, it is different in that the client- 
server embodiment shown in FIG. 140, In that Is designed to run on one computer system 14100, 
shown in FIG. 141. 

In this embodiment, when an applii::ation malces a call to the operating system to draw an 
element to a screen, the hooks placed in the OS dispatoh teble 14008 cause a corresponding draw 
routine within the programming 14010 of a scaled subpbcel-optimized screen generator program 
14006A to be evoked. This substitute draw routine draw a conesponding element to a virtual 
screen display list 10,206B. It also causes any portions of such screen eiemente drawn into the 
part of the virtual screen that fite within a view window 10210 see to be immediately displayed on 
the display screen 10220Aof the computer 14,100 by means of calling draw commands in the 
operating system, or by directly drawing to that screen themselves. 

When an applrcation program calls the operating system for a measure string conunands, 
that commands is likewise intercepted so the call returns font metrics for a substituted font size In 
the manner described above vtrith regard to functions 10608 through 10618 of FIG. 106A. 

A screen event input into the computer's scr^n is taken from the operating system's event 
queue and passed to an event position scaler, whtoh transfomfis the screen coordinate at which 
such an event was generated on the screen into a conesponding position In the virtual screen's 
layout represented by the display list, using the mapping of the view vrindow Into that virtual screen 
to control such a transfomiation. Once the coordinates of the event have been appropriately 
transfomied, the event Is returned to the operating system event queue so the operating system 
will respond to the event as if it have been entered onto the virtual screen. 

An embodiment of the invention of the type shown in FIG. 141 would allow a user of a 
computer to subpbcel-optimized, scaled-down, zoom, and perfonn selected text reflow upon 
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screens generated by standard computer applications 14002. even if they have not been designed 
to support such functions. 

In other embodiments of the invention not shown, the operating system of a computer can 
be modified to include functionality of the type shown in the scaled-subpixel-optlmlzed screen 
generator 14006A shown in FIG. 141. In yet other embodiments of the invention, application 
programs 14002, including browser programs, can be modified to support all or much of such 
functionality directly. 

FIG. 142 Illustrates how the embodiment of the invention shown in FIGS. 102 and 140 can 
be used to allow thin client computers, such as the thin client computers 200A through 200D 
shown in that figure, to be used to access Intemet content or application programs over wireless 
networlc. 

In this figure the computers 200A through 200D correspond to the thin client computer 200 
shown in FIGS. 102 and 140. The computer 200A is a handheld computer. The thin client 
computer 200B is a cellphone. The thin client computer 200C is a wristwatch computer. The thin 
client computer 200D Is a headmounted computer, or headmounted display for a porteble 
computer. Each of these client computers can have a subpbcel addressable display. 

At the time of the filing of this application It Is cumentiy possibte to manufacture screens for 
each of these types of devices having resolutions high enough for use by most aspects of the 
present Inventions. For example, at the cun-ent time it is possible to manufacture a 320 x 240 color 
LCD display with a diagonal measurement of 2 inches or less. Organic L£D devices can currently 
be manufadured with even higher resolutions. In the near future, the cost of such small screens 
should come down, and their availability and resolution should go up. 

All of the thin client computers shown In FIG. 142 have wiretess transcehrers which enable 
them to transmit and recehred infonnation of the type described above with a remote remote proxy 
server computer 210 of the type shown n FIG. 102 or a remote application server 14000 of the type 
shown above with regard FIG. 140. Such transceivers can be wireless LAN transceivers for 
communicating with a wireless LAN transceiver 14204 or digitel cellular wireless transceivers for 
communicating vdth a wireless Intemet transceiver 14202, or preferably a transceiver which has 
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been designed to oommunicate wKh both types of wireless transceivers. In other embodiments 
other types of vuireless communication, such as Bluetooth or Infrared communication, can be used. 

The remote computers 14,000AA through 14,000AC shown in FIG. 142 conespond to the 
remote server computer 14,000 shown in FIG. 140. 

The remote appGcation server computers 14,000AA shown in FIG. 142 represent laptop, 
desktop, server or other types of computers which can be programmed to operate as a remote 
application server computer 14,000. The subpixel-optimized application server 14,00OAB is a 
remote computer of the general type illustrated in FIG. 140 that is designed to run applications for 
a plurality of thin dient computers connected to a LAN or WAN associated with such clients. The 
remote computers 14.000AA and 14,000AB can communicate with thin clients over a private local 
area wireless transmitter 14,2004, or can communicate with them over the wiretess Internet as 
indicated by the numerals 10222 and 14202. 

The subpixel-optimized application server 14,000AC is an application server similar to 
server 14000AB, except that it is connected directly to the internet to allow multiple thin client 
computers 200 to use applications over the Internet by means of the wireless transmission network 
indicated by the numeral 14202. 

In FIG. 142 a proxy server 210, of the type described above with regard to FIG. 102. is 
shown connected to the LAN or WAN 14204. This, for example might be a proxy server intended 
to handle Web browsing which the Corporation wishes to keep off the internet it should be 
understood that other such proxy servers, such as those operated by companies providing 
commercial proxy serving services, would nonnally be connected directly to the Internet 10222 
shown In FIG. 142 as well. 

The system illustrated In regard FIG. 142 allows small computers that can be conveniently 
earned at virtually all fimes to access and display web pages and the output of most application 
programs. At the time of filing this application, the bandwidth of relatively inexpensive wireless 
LAN transceh^ers, such as the LAN transceiver 14204 shown in FIG. 142, is fast enough to allow 
thin clients of the type shown in FIG. 142 to view web content or the output of application programs 
almost is rapidly as one could view such digital content on a desktop computer connected to a 
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cable modem. And this is on a machine that can be canted one's poclcet. or on one's wrist, or as 
part of one's glasses, and which can be capable of accessing such media within several seconds 
after being tumed on. 

At the digital cellular bandwidth commonly available In America at the time of filing this 
application, it will normally talce several seconds to download the entire text of the most web 
pageSi and longer to download the web pages images. Of course many embodiments of the 
present Invention start to display text as soon as part of it is received, allowing the user starts to 
start seeing part of a downloaded page very quicidy. 

As of this filing new, higher speed, digital cellular systems have been developed that are 
capable of providing bandwidths In the range of hundreds of thousands or millions of bits per 
second. Once such higher speed systems become commonly deployed, users of the invention will 
be able to read and interact with web pages and applicafion screen on small, portable devices, that 
can be used within seconds of being tumed on most places they travel, with almost as much speed 
and convenience as if accessing them on a desktop or laptop through a DSL or cable modem 
connection. 

FIGS. 143 and 144 provide two views of a handheld computer 200A capable of functioning 
as a thin client for either proxy servers of the type described above with regard to FIG. 102 or a 
remote application server computer of the type described with regarcl to FIG. 140. 

In FIG. 143, the computer Is shown in the portrait orientation in which It has been designed 
for use. The native operating system on the computer is designed to draw fonts and graphical user 
interface elements in this portrait orientation. This is the manner In which many of the handheld, 
computers sold at the time of the filing of this application have been designed and built For 
example, there are multiple such handheld computers on the maricet today which have subpixel 
addressable screens with a 240 x 320 whole pixel resolution. Many of these computers also have 
subpixel striping which mns in a horizontal direction when the displays are in their Intended portrait 
orientation. 

Unfortunately, such a portiBit orientation does not provide the type of landscape aspect 
iBtio with which most people are used to using computers, and for which most web pages have 
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been designed. Furthennore, In the case where such computers have horizontal subpixel striping, 
such striping provide all of its potential increase In subpixel resolution In the vertical direction. 
Unfortunately, the display of text tends to benefit substantiaiiy more from an increase In horizontal 
resolution than It does from such an increase In vertical resolution. 

For all these reasons, many embodiments of the invention which use such portnait- 
oilentation machines are designed to use them when they been rotated by 90 degrees, as shown 
In FIG. 144, so they v\rill have a landscape aspect ratio more like that of the layout of most 
computer screens, and so that their subpixeis will provide an increase in horizontal resolution 
which is most useful for displaying text. 

The FIG. 145 is a highly simplified pseudocode nepresentation of how some aspects of the 
present embodiments can be used to respond to requests to draw basic shapes - such as 
rectangles, ovals, lines, and curves - using subpixel optimization. Such functionality can be used 
In applications of many different types, in operating systems, and in thin client software. 

In the example of FIG. 145, the pseudocode shown relates to a rectangle draw function 
14500, which could, among other uses, be used in place of the rectangle command 10918 
described above with regard FIG. 109A. Such a routine is evolced by a call to draw a rectangle 
which has its position, width, and/or height defined at higher resolution than the whole pixel 
resolution of a subpbcel addressable screen on which is to be shown. In response, the function 
14502 uses a subplxei-optimization routine to render the image of the rectangle defined at such a 
higher resolution, at subpixel resolution. This can be done using \rfrtually any subpixel optimization 
scheme, but for monochrome rectangles a bicolor optimization scheme, such as that described 
above will tend to provide the highest perceived spatial resolution. 

FIG. 146 is a highly simplified pseudocode representation 14600 of code 14602 which 
operates on a senrer and/or proxy computer and code 14604 which can be mn on a client 
computer, including a thin client computer, to allow applets downloaded from the server to draw 
subpbcekptjmlzed screen elements on the screen of the client 

In such an embodiment, a function 14,606 of the client requests media from ttie server. 
The server responds in function 14608 by downloading media, or data, including one or more 
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applet programs which can oin on the client computer. In function 14610 the client computer 
receives the media Including the applets, and function 14612 loads and runs the applets. In 
function 14614 the applets draw subpixel-opUmized elements to the subpixel addressable screen 
on the client computer. 

The applets can draw subpixel-optlmlzed elements either by copying or generating 
subplxekptimlzed bitmaps, by rendering text with subpixei-optimized fonts, or by drawing subpixel 
optimize shapes, such as the shapes of vector defined graphics or relatively simple geometric 
shapes, such as lines, rectangles, and ovals. 

FIGS. 147 and 148 illustrate how subptxel optimization can be applied to rollover images 
and GIFF animations, respectively. 

In the subpixel optimization routine 14700 shown in FIG. 147, both a non-rollover image 
14702, which is to be displayed when a pointing device is not detectably over the portion of the 
screen associated with the images, and a rollover image 14704, which is displayed when the 
pointing device is detectably over that screen portion, are both downscaled and subpixei-optimized 
by a function 14706. This produces a scaled subpixei-optimized non-rollover image 14708 and a 
scaled subpixei-optimized rollover image 14710. Then a function 14712 is used to select which of 
these two subpixei-optimized images is displayed based on whether the pointer is detectably over 
their associated screen area or not This makes the two subpixel-optimi^d im^^es act as a 
combined "rollover" graphic. 

In other embodiments of this aspect of the invention, a similar technique could be applied 
to two images that are associated with a button, one displayed when the button is not being 
pressed, and another displayed when the button is pressed. 

The method 14800 shown in FIG. 148 is similar to that described above with regard FIG. 
147. It takes each separate image 14802 through 148906 of a GIFF animations and subpixel- 
optimlzes it in a function 14808 to produce a oonresponding set of scaled-down, subpl)«l-optimized 
GIFF animations Images. Then function 14816 displays the subpbcelK)ptimlzed images in 
substantially the same manner that non-subpixekptimized gift animations are displayed. 
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The subpixel optlmizafions described with regard to FIGS. 147 and 148 can be used with 
other aspects of the Invention described above, Including In the accessing of web pages on a 
subpixel addressable screen, including those on thin client computers. 

FIG. 149 IHustretes a method 14900 for subpixel optimizing 3-D animation. This method 
Includes performing a set of functions 14904 through 14908 for each successive frame of the 
animation. 

Function 14904 nins a 3-D animation engine to create a bitmap of the ainent frame, or at 
least of those portions of the image which have changed since the last frame. This function 
generates such bitmaps at a resolution higher than the whole-pixel resolution at which the 
subpixei-optimized version of such bitmaps are to be displayed. 

Function 14906 then uses techniques, such as those described above, for scaling down 
and subpixel optimizing the frame bitmap, or at least changes made in the frame bitmap since the 
last frame. 

Next function 14908 displays the scaled-down, subpixel-optimized image of the frame 
bitmap, or at least of the changed portion of the frame, on a subpixel addressable screen. 

The method shown in FIG. 149 can be particular useful to allow people to play games, and 
see the images produced by such games at the higher resolution made possible by subpixel 
optimization. It can be used for such purpose on small screen, handheld devices. It can be used 
both vwth client computers displaying animated images generated on a remote computer, as well 
as with computers which are generating such animated images locally. 

FIGS. 150 and 151 illustrates one way in which the method of FIG. 149 can be used in a 
client server gaming application. 

FIG, 150 illustrates programming 15000 on a game server computer used in such an 
embodiment As indicated by the numeral 15002 and 15004, if the game server receives user 
input from one or more game client computers it sends Input to the game engine, if such input is 
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screen input, ft Is scaled appropriately to compensate for the difference between the user's scfBen 
resolution and the space which the game engine associates with screen inputs. 

In function 15008 the game engine computer computes a display list for the current frame, 
or for any changes associated with the cunent frame to a prior display list Then function 15008 
has a 3-D rendering program render a frame bitmap corresponding to the display list generated for 
the current frame, or render the changes required to the bitmap of the current frame. Such 
bitmaps are generated at a higher resolution than that of the subpixekpUmized Images which are 
to be created by the function 15010. 

If the client is generating different screen Images for different clients, the function 15008 
would be performed separately for each of those separate views. 

Next function 15010 scales down and subpbcel optimizes the cunrent frame bitmap or the 
bitmaps of cunfent changes to the frame* When the function is scaling down only bitmaps of such 
changes ft also comespondingly scales down the screen positions associated with those changes. 

Next function 15012 compresses the subpbcel-optimized bitmaps, and if appropriate, their 
locations, and function 15014 downloads the compressed, scaled, subplxel-optimized images and 
any such locations to the client for display. 

FIG. 151 Illustrates programming 15100 on a game client desfened for use with a 
programming of FIG. 150. 

Function 15101 receives downloaded images, then function 15102 decompresses them. 
Next function 15104 displays the scaled, subplxekpfimlzed animation frame bitmaps, or It displays 
bitmap of dianges over the image of the prior animation screen at the locations indicated for those 
changes. This Is done on a subpbcel addressable display. 

As indicated by numeral 15106 and 15108, when the client receives user input, it uploads 
that input to the game server with any screen coordinates associated with those Inputs being 
appropriately translated. 



wo 02/088979 



PCTAJS02/14380 



-104- 

In other embodiments of this aspect of the Invention the distribuUon of functionarity 
between the game server and the game cilent could be different, in some embodiments, a proxy 
server generally similar to that described above could be used to perfonn the subpixel optimization 
for display on a thin client of game content originally generated on a game server which is different 
than the proxy sender. In yet other embodiments the game client could itself perfomi the subpixel 
optimization. 

FIG. 152 is a highly simplified pseudo code description of an aspect of the invention which 
allows Images having associated transparency maps to be displayed with the subpixel optimizafion 
of both their fongmund image and transparency map 

The programming 15200 shown in FIG. 152 includes a function 15202 that produces a 
scaled subptxekptlmized bitmap of a foreground image, that is an image, the display of which on 
top of a bactcground or other prior bitmap is to be controlled by an associated transparency bitmap. 
The subpixel optimizafaon used can be either a bicolor or a multicolor subpixel optimization, or a 
combination of the two. Any method loiown for producing subpixei-optimized representations of 
images could be used, including those which have been described above. 

Function 15204 produces a subpixel optimbation of the Image's associated transparency 
map. Preferably a bicolor subpixel optimization is used» since a high resolution source image of a 
transparency map has transparency values that vary along a straight line In 3'<X)mponent color 
space, that of an alpha value ranging from 0 to 1 Such source image alpha values comespond to 
grayscale colors because, if the area of the transparency map source image corresponding to a 
given pbcel in the subpixei-optimized output image of that map Is covered by a unifonn 
transparency value, all of that output pixel's subpixels will tend to have equal alpha values. 
Preferably the bicolor subpixel optimization of the transparency map is createdusing the non-linear 
color balancing described above. 

Once such a subpixel optimization of a foreground image and its associated transparency 
map has been created, function 15206 displays this combination on a subpixei-optimized display. 
This process includes perfonning a loop 15208 for each pixel row of the displayed image, which 
includes a loop 15210 for each subpixel of each such row. The function 15210 causes function 
15212 and 15214 to be perfomned for each subpixel. TTie function 15212 sets the cun^nt alpha 



wo 02/088979 



PCT/US02/14380 



-105- 

value to the alpha value of the corresponding subpbcel of the subpixekptimized transparency map. 
Then function 15214 seis the luminosity of the current subpixel to the current alpha value multiplied 
by the luminosity of the conesponding subpixel of the subpixel-optimized foreground image plus 
the prior luminosity value of the current subpbcel in the background bitmap over which the 
transparency Image is being drawn multiplied by one minus the cunrent alpha value. 

This means that if the foreground Image is drawn over a prior bitmap, the extent to which 
luminosity of eadi of its separate subpbceFs is derived from the corresponding subpixel value of the 
foreground Image, or of the prior bitmap Is detemiined as a function of the con^sponding subpbcel 
alpha value of the subplxekptimized transparency map. 

In sorra embodiments of the invention images with associated transparency maps will be 
scaled and subpbcel-optimlzed on a server or browser computer, downloaded, and then displayed 
by function 15206 on a client computer. In other embodiments of the Invention, such subpbcel- 
optimized transparency Images will be made available on recorded digital media. In yet other 
embodiments of the invenfion they will be generated by the same computer that displays them. 

In other embodiments of the invention subpbcekptimized foreground images could be 
displayed using alpha values contained in a non-subpixel-optimized transparency map. 

In some embodiments of the invention lossy color compression will be used to represent 
groups of colors tiiat are perceptually close with one color. Such compression can be perfomned 
upon one dimentional transparency values, upon three dimensional transparency (l.o., opacity or 
alpha) values of the type described above with regard to FIGS. 60, 96, and 97, or upon color 
values having a transparency component value as an extra color dimension, as well as upon RGB 
component values. In such compressions, It Is generally advisable to prevent transparency values 
or component color valuesrepresenting an alpha one or zero, or values veiy close to one or zero, 
from being represented by transparency values further from one or zero, respectively. This Is 
because the eye is more sensitive to slight changes In opacity at the extremes of the transparency 
range than it is to such changes elsewhere in that range. 
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Subpixel-optimized images vrith transparency maps can be used on subpbcel optimize 
displays for all the purposes for wliich non-subpixel-optimized images are used with transparency 
maps. This includes use In animations and in web page layouts. 

FIGS. 153 through 162 are highly simplified pseudo code descriptions of aspects of the 
invention relating to subpbcel optimization of video and/or animation. Such subpixel opfimization 
can be used in the context of Web browsing as well as in virtually any other context in which video 
and animations is used. 

FIG, 153 represents programming 15300 used to subpixel optimize video represented 
using interpolation between video key frames. This programming includes a function 15302 that is 
used In the case where the video to be subpixel-optimized is received in compressed Ibmnat. It 
decompresses such video, so they can be subpixel-optimized. 

Function 15304 scales down and subpixel optimizes the keyframes of the video. Function 
15306 scales down but does not subpixel optimize interpolated changes between ke^mes. In 
some embodiments of the aspect of invention shown in figure 153, such interpolafion changes 
could be subpixel-optimized, but there is little benefit from doing so, since such changes appear so 
rapidly on a screen that their subpixel optimization would not be noticeable, and avoiding their 
subpixel optimization reduces computational overhead. 

Then function 15308 displays tiie scaled down video on a subpixel addressable display 
with the subpixel-optimized keyframes and tiie non subpixel optimize interframe interpolation. 

In ottier embodiments of tiie invention, tiiis concept of only subpixel optimizing portions of 
video that will be on tiie screen at one location long enough to be clearly perceived could be used 
inotiierways. 

FIG. 154 illustrates programming tiiat can be used to subpixel optimize video represented 
totally or partially by sequences of sub-whole-frame Image elements ttiat are to be drawn to a 
display frame. Commonly such video will also include whole frame images, and will use a 
sequence of sub-whole-frame draws to incrementally changes screen as needed to represent 
motion of one or more objects wtOiln IL This would Include animation of ttie type described above 
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wllh regaixJ FIG, 149. It can also Include various forms of video compression, Including video 
having keyframes and interframe interpolation of the general type described above with regard 
FIG. 153. 

The programming of FIG. 154 includes a function 15402 used where the video to be 
subpfocel-optimized Is received In compressed fomnat in which case that function decompresses It 
Next function 154004 scales and subplxel optimizes any frame images contained in the video, 
scaling them down by a display scale factor. Then function 15406 scales and subpbcel optimizes 
any change bitmaps, scaling both the size of such images and their location by the scale factor. 

Functions 15407 and 15408 repeatedly display on a subpixel addressable screen any 
scaled subpixel-optimized video frame in the video sequence. After the display of such a video 
frame it displays any of one or more scaled, subpbcel-optimized change bitmaps over the bitmap of 
that frame at the scaled position assodaied with that change bitmap by the function 1 5406. 

It can be seen that the method of FIG. 154 enables $ubpixel-optimi2»d video and 
animation to be drawn in a manner that reduces the amount computation required for subpixel 
optimization, since it does not require the subpbcel optimization of an entire frame each time a 
change is made to Its video Image. 

FIGS. 155 and 156 illustrate two different methods in which subpbcel-optimized Images that 
move relatWe to a frame can be displayed. 

FIG. 155 includes programming 15500 which displays an image with fixed subpixelation as 
it moves In whole pbcel increments relath^e to a larger image on a subpbcel addressable display, it 
includes a function 155002 which stores a subpixel-opfimized image, which can be produced by 
any method, including ttiose described above. It Includes a loop 15503 performed for each 
successive frame time. This loop comprises the function 15504 and 15506. The function 15504 
calculates a movement for the Image relative to the larger image. In this movement calculation the 
position calculated for the object at each display frame is rounded to the nearest whole horizontal 
and vertical pbcel location and the size and orientation of the image is not altered. The function 
15506 displays the image at the whole pbcel resolution location calculated for it by the function 
15504. Since only one subpbcel-optimized bitmap of the image has to be calculated, and that 
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Single image is repeatedly used as it moves across the screen, tills method is quite 
computationally efficient 

FIG. 156 describes programming 15600 which displays a moving Image with changing 
subpixelation. It includes a function 15602, which stores a high resolution source image of the 
image to be moved. It also Includes a loop 15603 perfomied for each successive frame time. Thte 
loop includes a function 15604, which calculates the cunent translation, rotation, and/or 
transformation of the high resolution source Image, If any for the current frame. Then the loop's 
function 15606 generates a scaled^lown, subpixel-optimized bitmap of the translated, rotated, 
and/or transformed bitmap so produced. This subpixel optimization talces into account the location 
of this transfonned bitmap relative to the subpixel anay upon which it will be displayed at a 
resolution higher than whole pixel resolution. Then function 15608 of the frame loop displays the 
resulting subpixel-optimized bitmap on a subpixel addressable display. 

Either of the methods ctescribed above with regard FIG. 155 or 156 can be u^d to display 
sprites in game animation, as well as animated text, or any other type of visual representation 
which is moved relative to a larger frame. 

The method of FIG. 155 tends to provide a less accurate representation of the motion of 
the visual object, but it is more computationally effidenL The method of FIG. 156 provides a more 
accurate visual represenbition, but is more computationaily expensive. 

in some embodiments of the invention a combination of these two methods could be used. 
For example, a small subset of possible mappings between the object and a subpbcel anray can be 
stored, and as the object moves it is displayed with that one of such stored mappings which most 
closely represents a higher resolution representation of its cument location relative to the subpixel 
array upon which it is to be displayed. 

FIGS. 157 and 158 illustrates aspects of the present invention used to optimize the display 
of DVD or an HDTV video by downscaling and subpixel optimizing such video for display on a 
subpixel addressable screen. This is particularly useful when used in conjunction with subpixel 
addressable screens that have a higher subpbcel resolution in the horizontal direcQon than they do 
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In the vertical direction, because both DVD an HDTV video commonly has an aspect ratio 
substantially wider than it is high. 

FIG. 159 illustrates aspects of the Invention that can be applied to video fonnats which 
represents subcomponents of video images as separate objects having different attributes. The 
particular example in FIG. 159 involves programming 15900 which subpixel optimizes the display 
of MPE&4 video. 

The programming shown In FIG. 159 includes a function 15902 that receives and 
decompresses and l\APEG4 video. It includes functions 15904 and 15906 that use different 
subpbcel optimization methods when scaling down different types of objects in the MPEG4 video. 
This function uses bicolor subpbcel optimizatton, preferably with nonlinear color balancing, on 
bicolor objects, and it uses multicolor subpbcel optimization on multicolor objects. Ifs function 
15908 display a combination of the bicolor and multicolor objects on a subpbcel^ptimlzed screen, 
moving such subpbcel-optimized objects relative to the screen as dictated by the MPEG-4 
description, using methods of the type discussed above vinth regard to FIGS. 155 and/or 156. 

Some aspects of the Invention are not limited to such use of different subpixel optimizing 
algorithms for different object types In the MPEG-4 data stream. But the use of such different 
subpixel optimizing algorithms can provide higher perceived resolution for bicolor objects, such as 
text, and thus has the advantage of providing a somewhat better Image. 

FIGS. 160 and 161 relate to systems in which users access subpixei-optimized video over 
a computer networlc. 

FIG. 160 illustrates programming 16000 used by a server computer that selves subpbcel- 
optimized. scaled down, video. Such a server could be a proxy server which accesses video 
requested by the client from yet another server computer and then downscales and subpixel 
optimizes it before downtoading to the client In other embodiments, the serving of such subpbcel- 
optimized video is perfomned without such an Intemiediary proxy server. 

The programming of FIG. 160 Includes a function 16002 which recehres a request for 
certain video from a dient computer. In many embodiments, such as the one shown In FIG. 160, 
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the request will also describe the horizontal and vertical subplxel resolution for which the video is to 
be subpbcel-optimized. In embodiments in which the server is only serving a set of clients having 
one fixed subpbcel resolution, such information is not needed as part of the request 

The function 16004 receives the requested video content This can be done by accessing 
it from a remote server, as described above; by accessing it from RAM or a mass storage device 
assodated with the' serving computer; by having such content dynamically generated; or by 
selecting a video fed from some source. 

Function 16006 scales down and subpbcel optimizes the receh^ed video to the subplxel 
rBsolution associated with the request of function 16002. Then fiinctfon 16008 compresses the 
subpbcel-opfimized video and function 16010 download that compressed video to the requesting 
device. 

The compression algorithm used for such subpbcekptimlzed images can include one 
which has a certain amount of ioss without substantially decreasing the Increased spatial resolution 
made possible by subpbcel optimization, as long as the location of the color values associated wth 
any pbcel In such subpbcel-optimized images are not moved in RGB color space by more than a 
relatively limited cobr distance. 

FIG. 161 describes a system 16100 that can be used vrith the aspect of the invention 
described in FIG. 160. This system includes proxy computer code 16100 and thin client computer 
code 16112, both of which arelDustrated by highly simplified pseudocode in FIG. 161. 

When the thin client receives a user request for certain video, function 16113 response by 
sending a request for the video, including the subpbcel resolution at which the video is to be 
displayed to the proxy. When the proxy receives the request for such video ite function 16100 
causes function 1 6103 to send a conesponding request for the video to a server from which it can 
be obtained. In many embodiments this will be a sen/er identified In the URL of such a request 

When requested video is received by the proxy server, function 16104 causes function 
16106 through 16110 to be perfonned. Function 16106 scaled down and subpbcel optimize the 
video to the subpbcel resolution associated with its request from the client; function 16108 
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compresses that subpixel-optimized video; and function 16110 downloads it to ttie client tiiat has 

requested It. 

When the client receives requested video iroiu the proxy, function 16114 causes funcWon 
16115 to decompress it, and function 16116 to display that downscaled. decompress the video on 
a subpixel addressable display. 

FIGS. 162 through 166 are used to illustrate how aspects of the invention can be used to 
Inoprove the appearance of digital ink. Digital Ink Is usually a black and white bitmap drawn on a 
screen in response to a user attempting to write or draw with his or her pointing device. In the 
past, digital ink bitmaps have usually been represented at a whole pbcel resolution in which each 
pixel is shown as either black, white, or in some devices a grayscale value. 

One aspect of the Invention is the use of subpixel optimizatton to represent digital ink with 
a higher resolution. When digital Ink which Is represented within the computer's memory by points 
and lines or curves between such points, the resulting mathematical de^ription of the lines 
between such points can have a much higher resolution than the whole pixel resolution of the 
screen. 

FIG. 162 Is a highly simplified pseudocode description of programming that can be used to 
optimize the clarity with whtoh digital Ink can be viewed. 

The digital ink code 16200 shown in FIG. 162 Includes a function 16202 which responds 
to user input with a pointing device while In digital ink draw mode, by recording the strokes of the 
pointing device as a series of points and curve or lines between such points. Function 16206 
draws Ink on the screen using a subpbcel optimization of the lines and curves. This can be done 
with virtually any subpbcel optimizatton scheme, but it is prefeirabiy done with a btoolor subpixel 
optimization scheme, such, as for example, a bicolor subpixel optimization scheme using non- 
linear cotor balancing. 

FIG. 163 illustrates some digital Ink 16302 which has been drawn on the screen of a 
handheld computer 16300. Because this illustration Is printed \Mth a printer which can only 
represent whole pixel luminosity values, the digital ink illustrated In FIG. 163 displays subpixel 
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optimizatlon as grayscale antnaliasing. It to be appreciated that when viewed on a subpixe! 
addressable display the image would appear even more dear than shown In FIG. 163. 

If the user of the digital inic programming selects to scale up a representation of a portion 
of digital ink, function 16208 causes function 16212 to produce a subpixel-optimteed bitmap of the 
digital Inic lines and curves, using a bicoiorsubpixel optimization with non-linear color balancing, at 
the user selected scaied-up size. Then function 16212 displays that scaled-up image on the users 
screen. 

FIG. 164 illustrates a scaled-up representation 16302A of the portion of digital \vk 16302 
shown in FIG. 163. This provides a substantially more clear representation of the digital Ink, than 
is produced by merely blowing up the pixelation of fiie digital ink's representation 16302 shown in 
FIG. 163, as is illustrated by ttie bitmap 16302B shown in FIG. 165. 

It should be noted that the bitmap shown in FIG. 165 is actually more pleasant to look at 
than somescaled-up representation of digital ink because the bitmap shown in FIG. 163 has been 
printed with whole pbcel grayscale values with anti-aliasing, which is not used in some digitel ink 
representations. 

If the user selects to scale down the representation of digital Ink, function 16214 causes 
function 16216 to produce a subpixel-optimized bitmap of the ink's lines and curves using bicolor 
subpixel optimization vwth non-linear color balancing at the selected scaled-down size, and then 
causes function 16218 to display that scaled-^lown bitmap on the subpixel addressable display. 
The results of such a process is illustrated by the bitmap 16302C shown in FIG. 166. 

These aspects of the invention can be modified to deal v\nth digital inic that has been 
recorded as whole pixels that are either on or off. This can be done by having a routine estimate a 
centerline of each stroke represented by such "on" pixels, and then producing a subpixel opt'mize 
image of the digital centerline at various scales as described above. An more accurate but 
more computationally expensive approach would be to seek an optimal fit between successive 
portions of such digital ink and a conesponding succession of lines and curves, such as, for 
example Beziercun^es. 
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In other embodiments, subpixel optimization could be perfomied on bitmaps which have 
been produced by digital ink drawing by merely performing subpixel-optimized scale ups or scale 
downs upon such bitmaps. 

Some embodiments of the invention that relate to digital ink could be used with non 
subpixel-optimized displays, by replacing subpixel optimization with grayscale anti-aliasing. 

FIG. 167 illustrates components that can be Included In many of the server, client, proxy 
server, thin client, remote, desktop computer, or other computers refen^ to above. It should be 
understood that not all of the components shown In FIG. 167 will be in all such computers, and 
most such computers will Include other components besides those shown In FIG. 167. 

This figure Is provkJed to make clear that most of the computers used with various aspects 
of the present Inventton include some type of processor 16716 capable of executing programming 
16702 to cause it to perform the functions of such aspects of the invention and to read and write 
data 16704 according to the methods of such aspects. The present invention relates to not only to 
methods but also to such computer programming and data, as well as to computer systems which 
have been programmed and/or hardwired to perfomi such methods or to use such data. 

In most such computers the invention's programming will be stored in RAM 16706; ROU 
16707; or a mass storage devfce such as a hard drive 16708, floppy drive 16709. CD-ROM drive 
16711, and/or DVD drive 16713. It can also be stored in nnachlne-readable media, such as on a 
floppy disks 16710, CD ROMs 16712, DVD ROMs 16714, or virtually any other type of machine 
readable storage media. The invention's programming and/or data can also be the represented as 
propagated signals Indicated by the numeral 16719 whteh can be received by the computer 
through some sort of communication port, such as the networic interface 16720. 

FIG.FIG. 168 provides a whole-pixel grayscale representation of a 320 by 240 screen 
showing a small subpixel-optimized font produced using the non-linear color-balance method 
described above with regard to FIG.FIG. 60 through 97. This figure is Mentteai to FIG.FIG. 56 
except that a portion of its text is encircled by dotted lines 16800. 
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FIG. 169 Is an eight times blowup of the portion of the bitmap shown in FIG. 168 within the 
dotted lines 16800. It shows that nwst of the vertical strokes in the font shown in FIG. 168 contain 
color-balance distributions one their lefthand side which blur the clarity of such fonts. 

One of the major benefits of tt)e non-linear color-balancing method of producing subpbcel 
optimized font bitmaps is Its ability to decrease the blurring of character-font shapes by the non- 
linear method with which it seeks to substantially prevent the distribution of color balandng values 
where it is not needed for cotor balance. 

Upon observing the spreading of color values to the left of the main strokes of fonts of the 
type shown In FIG. 169, the inventor of this aspect of the invention sought to see If such spreading 
could be reduced. He tried to detemvne what the source of such spreading was. 

Refening now to FIG. 170, he found that the algorithm used for creating non-linear color- 
balanced bitmaps was designed to automatically place two padding columns of subpixels 17000 to 
the left of the leftmost subpixel column 1 7002 in the rasterization of a character-font shape that 
included an actual non-zero coverage value 17004 (i.e., was actually covered by a portion of the 
character-font shape being represented by the rasterization). This was done to provide room for 
the spreading of color balancing color values into the two subpixel column to the left of the leftmost 
subpixel column containing such a non-zero coverage value, if such a leftward spreading was 
required by the non-linear color balancing algorithm, described above, which allows color balancing 
distribution two pixel to the left of a totally or partially covered subpixel.. 

Unfortunately padding the rasterization subpixel anay with only two such subpixel columns 
17000 tends to have the undesirable effect of making the leftmost subpixel column 17002 that 
contains such a coverage value be the rightmost subpixel column of the pixel column containing 
the two padding subpixel columns. In an RGB display this would cause the leftmost subpixel 
column containing an actual coverage value to comespond to a blue subpixel. 

This is undesirable because it tends to cause pixels In the leftmost pixel column in a font 
bitmap to contain two leftmost subpixels which have no actual coverage value and a rightmost 
subpbcel which does Include a non-zero coverage value, requiring that non-zero coverage value to 
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be distributed to achieve color balancing. This is a reason for much of the leftward blunring of 
major vertical strokes shown in FIG. 169. 

The inventor noted that character-font shapes hinted with systems that allowed boundaries 
of vertical stroi^es to be positioned in increments finer than the width of a subpbcel column had 
often been designed by the individuals who hinted them to start the leftmost edge of their leftmost 
vertical stroke, such as the edge 17100 shown in FIG. 171 only a slight distance Into the leftmost 
subpixel column containing non-zero coverage values 17002. This would substantially reduce the 
amount of the non-zero coverage value contained within the subpbcel column 17002 which had to 
be distributed by non-linear color balancing, thus greatly reducing undesirable blurring in the 
subpixel optimized representatton of the character. 

For example, the Inventors found that many of the best hinting combinations, when used 
with such algorithm, cause the first vertical strokes of a character, such as the vertical strokes 
17102 shown in FIG. 171 to have Its leftmost edge slightly into one subpbcel column, with total 
coverage In three successive subpixel column to the right, so as to cause the second leftmost pixel 
column 17103 in the resulting bitmap to have one or more pbcels totally covered so as to require no 
color-balance spreading. 

In such an optimized hinting process, subsequent vertical strokes would be aligned to 
cover three adjacent subpbcel column starting at a distance of three, sbc, or nine subpbcel column 
from the rightmost edge of the first vertical stroke. This would cause the subsequent vertical 
strokes, such as the vertical strokes 17104 and 17106 shown figure 171, to have multiple pbcels 
that are totally covered, so as to require no color balandng. 

Alttiough fonts of the type shown in FIG. 168 to 171 are more readable than most 
subpbcel-optlmlzed font bitmaps produced by prior art method, as a result of these investigations 
the inventor has figured how to produce even more clear subpbcel optimized font, as shown in figs. 
172 through 174. 

FIG. 172 shows a whole-pbcel grayscale bitmap representing a subpbcei-optimized 320 by 
240 pixel display of a web page of the type shown in FIG. 168, except that it uses a new, more 
dear method for produdng and displaying font bitmaps. 



wo 02/088979 



PCTAJS02/14380 



-116- 

RG. 173 shows a four times blowup of the portion of FIG. 172 shown in the dotted box 
numbered 17200FIG.. 

FIG. 174 shows a liirflierfbur times blowup of the portion of text shown In the dotted lines 
17300 In FIG. 173. 

As can be seen from tooldng at FIGS, 172 through 174. there is relatively little horizontal 
spreading of color values from many of the vertical strokes contained in the font bitmaps shown In 
those figures. It should be noted that the unifbmi light gray background in FIGS. 173 and 1 74 
resuHs because the text in those figures was taken from a portion of the web page of FIG. 1 72 that 
had a background color, not because of any spreading due to cotor balancing. The fonts shown In 
these figures are substantially more clear than those shown in FIGS. 168 and 169. 

The Inventor has made this improvement by aligning the leftmost edge of a character's 
leftmost vertical stroke with the left edge of a pixel boundaiy. In many embodiments this is done 
by inserting three padding subplxel column 17500, shown In FIG. 175 before the leftmost subpixel 
column that contains a non-zero coverage values. This automatJcallyaDgns the leftmost 
i^rization unit p.e.. subpixel) all or partially covered by a character's ouUinevnth the leftmost 
edge of a pixel column. If a characters is hinted so its leftmost outline edge is aligned vwth the 
leftmost edge of a rasteilzatton unit, this will automatteaOy cause that leftmost outline ec^e to be 
aligned with the leftmost edge of a pixel In the resulting font bitmap. When the leftmost edge of a 
font outline Is a vertfcal sboke this makes it very easy to create a font bitmap that has dear 
leftmost vertk»l edge, even after non-linear cotor balancing. 

FIG. 176 shows one of many possible hinting interfaces that can be used vinth the present 
Invention. In this hinting Interface the dotted lines 17602 Is a line that can be moved by the user to 
interactively define the left sWe bearing for a desired character. The dotted Ine 17604 Is a 
movabte line that defines the right side bearing. The left side bearing is the distance between the 
initial reference point sometimes called the pen position, relative to whfch a character Is to be 
drawn and tlie leftmost edge of the bitmap of the character being drawn. The line 17604 
corresponds to the location relative to the bitmap at which the pen posifion will be nomraliy be 
placed at the start of the drawing of the next success diaracter along a line of text The right 
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Side bearing is the distance between the ilne 17604 and the rightmost edge of the bitmap of the 

character being drawn. The advance width is defined as the distance between the lines 17604 and 

17602. This represents the nomiai totai width between pen positions before and after the drawing 

of a character's bitmap. In some embodiments, the ieft side bearing value and the advance width 

are rounded to whole multiples of pixel widths, although in other embodiments this need not be 

true. In some cases, the left and/or right side bearing values can be negative. For example this 

often happens with italic fonts in which the bitmaps associated with successive characters often 

overiaps portions of each othei's advance width. 

Each of the small rectangular dots 17606 shown in FIG. 176 correspond to the center of a 
rasterization unit which, In subpixel-optimized font bitmaps, correspond to an individual subpixel. 
In this particular hinting Interface asterizatlon units more than half covered by a character-font 
shape's outline are shown in black, although In more advanced internees such rasterization units 
could be shown with grayscale coverage values. The character-font shape's outlines are shown In 
the FIG.figure and each point which defines a segment In the outline is numbered, whether it be a 
control point or a segment endpoint. 

FIGS. 177 through 181 are used to help explain some of the steps described in the highly 
simplified pseudocode contained In FIG. 182. 

FIG. 182 is a highly simplified pseudocode description of programming 6000A, which 
corresponds generally to the pseudocode shown in FIG. 60, except that the pseudocode shown In 
FIG. 182 focuses on computational aspects which relate to the Improved method of producing 
more clear nonlinearty color-balanced subpixel-optimized bitmaps described above with regard to 
FIGS. 172 and 176. 

The pseudocode Includes a function 18202. which detemilnes the tightest rectengular 
anay of rasterization unKs Into which a character-font shapes can be placed, taking into account 
the alignment of Its shape relative to such rasterization units defined by its hinting. 

The position of the font outline relative to the individual rasterization units In which it occurs 
is not changed by this function. Thus, if such an outline's leftmost point occure other than at the 
left edge of the rasterization unit it is in, that rasterizatton unit will appear at the leftmost edge of the 
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tightest rectangular array produced by ftinction 18202, aiKl the leftmost point of that outline would 
occur within the leftmost rasterizafion unit column of «iat rectangle, Ixjt It would not occur at the 
leftmost edge of that iefimost cobimn. 

FIGS. 177 and 178 are used to help explain this function. FIG. 177 corresponds to a 
hinted character-font shape outl&ie. FIG. 178 shows the rectangle ofrasterizatlon units (each 
corresponding In size to a subpixel) returned by function 18202 for the character outline shown in 
FIG. 177. This grid conesponds to flie tightest, or smallest, rectangle Into which the rasterization 
unit containing tiie character font shape fits. 

Once the funcdon 18202 has been completed, functions 6002A through 6006 are 
peribrmed. These correspond to steps 6002 through 6006 of FIG. 60. They are used to determine 
a coverage value for each rasterlzatton unit contained In the rectangte returned by function 
18202FIG.. Each such coverage value represent the percent of the subpixel covered by the 
higher resolution character-font shape outihe being rasterized. 

FIG. 179 Iflustrates the coverage values calculated for each rasterization unit in the array 
shown In FIG. 178. In it cover^e Is represented by the percent of the rasterization unit that is 
cofored black. In FIG. 179 the portion of the resulting bargraph In each rasterization unit 
representing coverage is placed at the top of that unit if the corresponding part of the unit covered 
by the character-font shape outline occurs at the top of the rasterization unit 

In FIG. 180 the bargraphs for all individual rasterizatbn unite are placed starting at the 
bottom of the conesponding subjHxel unit, so as to mala them coniespond more closely with the 
representation of coverage values shown In FIGS. 46 through 52 and 92 through 93, described 
above. 

Once the character-font shape has been rasterized, step 18204 maps the resulting array 
of subpixel coverage values into an anay of subpfacei-addressabte pbcels. It does so aligning *e 
first column of rasterization units in toe tight rectangle describal above with the leftmost subpbcel of 
a pixel row. This causes ttie leftmost column of rasterization untts which have a nonzero coverage 
value to be placed as a leftmost subpixel column In a whote pi)ffil as described above with regard 
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to FIG. 175.FIG. In the example lllusted in FIGS. 177 through 181, this causes the resulting 
subplxel array to appear as shown in the central pixel columns labeled 18102 in FIG. 181 . 

Next a step 18206 pads the bitmap an^ay being created for the current character with a 
pixel column comprised of three subpixeFs to the left of the pixel containing the leftmost subplxel 
column containing an actual nonzero coverage value. This causes the subplxel an^y in the 
example to appear as shown by the combination of pixel columns 18104 and 18102 in FIG. 181 . 

Next a step 18208 pads the bitmap array with two or more subplxel columns to Its right so 
as to cause the total number of subpbcel colunuis of the bitmap to be an even multiple three, that is 
to be an even number of whole pixel columns. This causes the exannple subplxel array to appear 
as shown by the combination of pbcel columns 18104, 18102, and 18106 In FIG. 181. 

Step 18210 adjusts the left and right side bearing value to compensate for the addition of 
the padding pbcel columns. Thus, for example, a bitmap which would othenArise have a left side 
bearing of one pixel width would be changed to have a left side bearing of zero to compensate for 
the addition of the left side padding column. SImilariy a bitmap which had and extra pixel column 
added to its right side would decrease its right side bearing by one pbcel width. 

Next function 18212 perfomns non-linear color balancing, which in many embodiments will 
correspond to the steps described by the loop 6008 shown in FIG. 60, described above. 

Once this has been done, in embodiments using a packed color value representation of 
0ie type described in FIG. 96 above, step 18214 converts the pbcel color values resulting after the 
color balancing operation into corresponding values from a more limited color palette. 

Note that the methq^ of FIG. 182 allows room for any color balancing which might be 
necessary, wtthout tending to cause the unnecessary color spreading discussed above with regard 
to FIGS. 168 and 169. It does this by insuring that there are at least two subpbcels to the left and 
to the right of any subpbcels conesponding to area covered by the font shape being rasterized. 

In other embodiments of this aspect of the invention other methods will be used to cause 
leftmost and rightmost edges of font shapes and vertical strokes to be aligned with whole pixel 
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boundaries, so as to take maximum advantage of the capabBity of non-linear color balancing to 
reduce smearing. In some such embodiments, vidiether or not a padding pbcel columns was added 
to the left or right side of a font bitmap could be a function of whether or not color balancing 
distributions were required in sadn columns. 

FIG. 183 describes functions for drawing a string of characters using the bitmaps produced 
by the method described In FIG. 1 82. Tliis pseudocode is similar to that described above with 
regard FIG. 97. except that It focuses on an aspect of the Invention that Is quite useful with the 
method for producing more dear non-llneariy coior-baianced subpixek)ptlmlzed font l»lmaps 
describe with regard to FIG. 182; 

When the draw string function 18300 shown In FIG. 183 Is called, a step 18302 sets the 
pen position to a start position specified by the draw string caB that Indicates where the display of 
the string is the start 

Then a loop 9714A similar to the loop 9714 described in FIG. 97 Is perfomned for each 
character the string to the display. 

In this loop a step 9716 accesses the cunent character^ font bitmap. Then a step 18304 
sete the character start position to the current pen position. Then a step 18306 adjusts the cunent 
pen position by the left side bearing. As has been described above, the left side bearing has been 
changed from what It would nomiaily be to talte into account the fact that the character bitmap has 
been padded with one extra pixel column on its left hand side, and thus wlB be decreased by the 
width of one pixel column. 

Next a step 971 8A Is performed for each pixel in the font bitmap. This Includes a substep 
18308, which teste to see if the cunent pixeFs value is nonzero. If so, it draws the pbcel on the 
screen at a position detennined as a function of the current pen position. 

If the current pixef s value Is zero, it represenb a toteliy transparent pbcel, meaning the 
background color previously at the posifion of the current pixel should be left unchanged. In this 
embodiment of the Invention the functtons described in FIG. 96 reserve the value 0 to represent 
such a toteily transparent pixel. 
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Thls practice of not writing transparent pixePs Is applied to all pixels of the bitmap in the 
embodiment described In FIG. 183. This practice Is particular valuable with regard to pixels in the 
padding column placed at the left most edge of a character-font bitmap by step 18206 described 
above regard FIG. 182. This is because pixels In such padding columns will commonly have no 
color values spread Into them as a result of non-linear color-balancing when verb'cal stroke 
boundaries have been aligned to vertical pixel boundaries. As a result, such pixels will be 
transparent and color values which may have been placed In their location by the character to Its 
left can remain unchanged, allowing the pixel columns of adjacent characters which contain 
coverage or color balancing Infomnation to be placed adjacent to each other. 

This can be seen for example at the location indicated by the numeral 17302 in FIG. 173 
where the pixel column between the V and V of the word "Web" contained color values from the 
V which have been allowed to show through the transparent, and thus non^written, left side 
padding column associated with the "e". This can also be seen at the location indicated by the 
numeral 17402 shown in FIG. 174. in which the pixel column between the Y and the V contain 
color values from the Y which are not overridden by the transparent padding pixel column of the 
V. 

As those skilled in the art will recognize, function 9718A will require some sort of iteration 
controlling the position at which pixels are drawn to be repeated for each row of a font bitmap, so 
as to have each of Hs bitmaps drawn in the proper place. 

It should be appreciated that in other embodiments of the inventton, functbn could be 
prx)vided which would allow overiapping non-transparent pixel values from adjacent characters to 
be combined, rather than merely allowing non-transparent cotor values from one character to show 
through when the conBspondIng pixels of the following character are transparent. 

Preferably such a process would allow combination of such transparency values on a 
subpixel-by-subpixei basis. Such a process could provide an even more accurate representation 
of closely spaced letters, although it would require nnore computation. 
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One way of achieving this result would be as follows: Add each of the three corresponding 
alpha component values associated with any overlapping pixel between characters, clipping any 
component values at their maximum possible value. And then drawing each of the resulting pixels, 
using the combined component alpha values to detemdne how much foreground color and how 
much background cotor should be drawn at Its location. 

FIG. 184 Illustrates an alternate embodiment of this method for providing more clear non- 
linear color-talanced font bitmaps. It Illustrates a hinting Internee similar to that described above 
regard FIG. 176, isxcept that It includes an Interface feature 18402 comprised of a user-movable 
line or control. This control allows the user to selectively position, relative to his or her character- 
font shape outline, the Ideation to be aligned with the leftmost edge of a pixel column following the 
leftmost padding pbcel column. 

Such an interface feature Is more desirable when hinting fonts which have a leftmost edge 
which is other than a vertical stroke. For example, when dealing with a character-font shape 
having a leftmost main vertical stroke with a small serif sticking out from to its left edge by less than 
a full pixel width, the hinter may want to have the main leftmost edge of the vertical stroke aligned 
with a whole pixel boundary, rather than the more leftward serif. The interface feature shown In 
FIG. 1 84 would make such an alignment easy for a hinter to select. 

Another way of giving a hinter the equivalent capability would be to allow him or her to 
select vi^etherto add only two subplxel padding columns, as described above with regard FIG. 
170 or 171, or to add three or more such subplxel padding columns, as is described above with 
regard to figs. 175, 181 and 182. 

The just described method for making non-linear color balanced subplxel optimized 
bitmaps more clear Is not only applicable to small fonts of the types shown in figs. 172 through 174 
but also to larger fonts, such as the relatively large font shown In FIG. 55. 

It should be appreciated that subpixel optimization can usually represent a font bitmap with 
just three different types of pixels: a foreground pixel, a background pixel, and an inlemiediary, 
color balancing, pixel. A foreground pixel represents a portion of the font image totally covered by 
the font shape being represented, and Is drawn with the foreground color with which the character 
Is being represented. A background pixel represents a portion of the font image totally uncovered 
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by the font shape, and is dfBwn with the coior of the baclcground on top of which the font is being 
shown. An intennediate pixel represents a pixel that is partially covered by the font shape and/or 
which receives color balancing distributions for a nearby pixel. The color of each of its subpixel's is 
detemiined separately by color balancing. 

When prior art Finear color balancing of the type described above with regard to FIGS. 46, 
47, 52, and 93 are applied to fonts, color balancing is performed across every edge of a character 
shape in the direction of subpixel color variation, even if that edge Is perfectly aligned with a pixel 
boundary. This leads to the spatial smearing of the shape of ail letters, no matter how well hinted. 

When non-linear color balancing of the type described above with regard to FIGS. 48, 49. 
51, and 91 is applied to fonts, hinting can be used to greatly reduce the spatial smearing caused 
• by cofor balancing. In portions of a character's shape where its edges are aligned witii pixel 
boundaries, often no color balance distribution will be required across pixel boundaries. This is 
because such non-linear color balandng only distributes color imbalance ttiat occurs within a give 
pixel. This alfows foreground pbcels to be next to baclcground pixels along ttie direction of subpixel 
color variation In such locations, greatiy increasing ttie perceived clarity of tiie font shape. This is 
shown in FIG. 173 and 174 in which substantial portions of ttie vertical strolces in tiie 8 pixel per em 
font shown ttiose figures have been hinted so fliat ttie edges of ttiose strolces align witti pixel 
boundaries. As a result, foreground pixels are located horizontally next to bacl<ground pixels along 
substantial portions of ttie edges of many such vertical strolces. Even witti ttie less optimal hinting 
of leftmost vertteai strolce edges shown in FiGS. 168 and 169, ttte amount of color-balance 
smearing is substantially less ttian ttiat which would result from prior art linear color balancing. 

FIGS. 185 ttirough 190 are highly simplified pseudocode descriptions of user interface 
innovations tfiat can be used to improve ttie browsing of Web pages, particuiariy when such 
browsing is perfomied on relatively small or relatively low resolution screens. 

FIG. 185 is a higher level description of tiie selected-text reflow method described above 
witti regard to FIGS. 129 tiirough 134. This metiiod 18500 includes a function 18502 tiiat 
accesses a Web page's content and a function 18504 tiiat performs a first layout of ttie Web 
page's content placing text at different horizontal locations indicated for text in ttie web page. The 
maricup languages used to describe Web pages have multiple mettiods of indicating that different 
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portlons of text are to be drawn at different horizontal locations or in different horizontal ranges In a 
web page's, including, to name Just two, the use of tables and frames. 

Once such a layout has been perfomned, function 18506 displays the elements of the 
layout at a given scale and at relative positions detennined by the first layout After this display 
has been perfbnned a step 18508 enables the user to select a portion of the text at a given 
horizontal location In the display of the first layout On way of enabling this Is described above with 
regard to FIG. 130. 

If such a selecfion Is made, function 8510 causes function 18512 and 18514 to be 
perfomied. Functton 18512 perfbmis a second layout of the text that has been selected by the 
user. This second layout reflows the selected text across the lines of the new column in which the 
text has a different ususally larger, font size relative to the width of the lines in that column. When 
this second layout is been perfbmned, function 18514 displays the layout of the new column at a 
scale that fills at least two thirds of the vridth of the screen or screen window on which the web 
page is being displayed. 

As indicated above with reganl to FIGS. 135 through 137, the second layout in such 
selected-text reflow method allows a user to see selected portions of the Web pages layout in at 
large easy read font sizes. This can be a tremendous advantage on both low resolution screens, 
screens that are small, and/or screens that are relatively far from their viewer. The first layout in 
such a method allows the user to get a view of how the web page is intended to look In more 
nomial displays, and allows the user to more rapidly select which portions of the text he or she 
desires to see le-displayed at a larger fr)nt size. 

FIG. 186 is a high-level pseudocode description of a zoom-to-fIt method 18600, of the 
general type described above vrith regard to FIGS. 118 through 120. 

This method Includes a function 18602 that accesses a Web page's content and a 
function 18604 that lays the Web page's content out 

Once such a display of the layout is being shown on a screen, function 18608 enables the 
user to drag a pointing device across this display. During such a drag, If the dr^ continues across 
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a boundary associated with a screen edge, a function 18610 causes function 18612 to scroll, onto 
the screen, portions of the layout that were previously off screen on the other side of the screen 
edge. This Is done to allow user to select with a drag a portion of the layout that is either too large 
to entirely fit on the screen at the cunent display scale or that was positioned at the start of a drag 
so that only part of it was on the screen. 

If the user releases the drag, function 18614 causes functions 18616 and 18618 to be 
perfonned. The first of these causes a part of the layout to be defined as selected based on the 
positions in the layout that corresponds to the start and end of the drag. Such a selected part can 
comespond to a portion of the layout having the horizontal or vertical range of the drag or to an 
area having diagonal comers corresponding to the start and end of such a drag. Then function 
18618 displays the selected part of the layout at a scale that causes it to substanfialty fit the 
screen. 

FIG. 187 is a high^evel pseudocode description of a drag scroll method 1 8700 that allovi^ 
a user to easily navigate within the display of a web page's layout 

This method includes a function 18702 that accesses the Web page's content, a function 
18704 that performs a layout of the Web page's content, and a function 18706 that displays all or 
portion of that layout at a given scale factor. Then a function 18708 enables the user to drag a 
pointing device across the display of the layout. Function 18710 responds to any such drag across 
a boundary associated with a screen edge by scrolling onto the screen, past the screen edge, 
portions of the layout previously offscreen. 

This method can be used as part of, or independently from, zoom selection functions. It 
has the advantage of enabling a user to scroll around the display of the layout of a web page by 
merely dragging a pointing device across a boundary at, or near, an edge of the display screen. 

FIG. 188 Is a high-level pseudocode descripflon of a click-zoom method 18800 that 
enables the user to rapidly selected to zoom in on a desired portion of the display of a layout of a 
web page. This method includes a function 18802 that accesses the web pages content, a 
function 18804 that perfonns a layout of the Web page's content, and a function 18806 that 
displays all or a portion of the Web page's layout at a first scale. A function 18808 enables the 
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user to dick appointing device at a selected iocation in tlie display of the layout at the first scale, 
and function 18810 responds to such a click by perfonning a zoomed-ln display of the portion of 
the layout around 0ie location in the layout at which the click was perfonned. Commonly the 
zoomed-in display will be centered on the tocatlon In the layout at vrtiich the click was made. 

FIG. 189 is a highly simplified pseudocode description of the zoomciick method 18900 
described above in some detail with regard to FIGS. 121 through 128. 

This method includes a functbn 18902 that accesses the Web page's content, a function a 
18904 that perfonns a ^yout of that content, and a function 18906 that displays all or a portion of 
the web page's layout at a first scale on a display screen having an associated pointing device. In 
the particular embodiment of this method described in FIG. 189, the screen is a touch screen and It 
is intended that the pointing device can be a person's finger. 

Once the display of the layout at the first scale has been perfonned, a function 18908 
responds when a press has been made to the touch grain display. When such a press occurs, this 
function causes functions 18910 through 18922 to be perfonned. 

Functk)n 18910 replaces on the screen the display of a portion of the web page at the first 
scale vrith a zoomed-in display of a portion of the web page at a larger scale. This zoomed porijon 
includes a selected location In the layout associated with touch screen press. Preferably the 
selected layout position has substantially the same iocation on the screen In the zoomed-in display 
as it had In the display at the first scale at the time of selection. By substentiaily same position, It is 
meant that the selected positioned should have k)cafions on the screen both immediately before 
and after the zoom that appears to con^spond to the same touch positioned on the screen. 
Preferably this would mean that the change In the selected positions screen location would not 
change by more than twenty percent of the width or height of the screen immediately after such a 
zoom. 

Once the zoomed-in display is shown, fiinc^on 18912 displays a cursor above the location 
at whbh the screen is being touched to Indicate the selected iocation In the web page layout 
assodated with the touch. In some touch screen devices, particulariy those designed for use with 
styluses having relatively fine points, there is no need for such a cursor, since the user can see 
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vwth considerable accuracy the point at whidi the screen is being touched. But in touch screens 
designed Ibr use with fingers as pointing devices it often desirable to place a cursor above the 
location at which the screen Is being touched so the user can accurately see the location In the 
screen's display that Is associated wrfth such a touch. This is particularly desirable when the 
method is being use with a display, such as that shown in FIGS. 121 through 128. that is relatively 
small compared to the size of a human finger 

During the continuafion of the touch a function 18914 responds to any movement of the 
touch by correspondingly moving the cursor in the zoomed display. Also during the continuation of 
such a touch, a function 18916 response to any movement of the touch across a boundary 
associated with a screen edge by scrolling onto the screen, past the screen edge, portions of the 
layout at the zoomed scale that were previously off the screen. This allows the user to rapidly and 
conveniently scroll vwlhin the zoomed display of the web page while in zoomclick mode. 

Functton 18918 responds if the user releases a touch at a given positioned in the zoomed 
display of the web page. If so. a fandion 18920 acts as if a pointing device cOck had occuned at a 
positioned in the web page conesponding to that of the release. For example, if the release is at a 
layout location conresponding to a web link, the system will respond by selecting the link, or if the 
release Is at the location of a radio button, the system will respond by flipping the stale of the radio 
button. 

Once this has been done, a funcHon 18922 replaces the display of the zoomedHn layout 
on the screen with a display of the layout at the same first scale factor at which the web page was 
displayed before the pointing device press was detected by iuncOon 18908. 

As described above with regard to FIGS. 121 through 128, zoomdick provides a valuable 
technique for allowing a user to rapidly see and select desired portions of a web page at a 
zoomed-in scale that makes the contents of those selected parts easier to read and easier to 
accurately select with a pointing device. 

FIG. 190 is a highly simplified pseudocode description of a method 19000 that allows a 
user to see a zoom-out view of a web page using greeking to represent text lines. GreeWng is the 
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representation of the size at which porbons of text are laid out in a document by non-readable 
graphic representations. 

This method includes a function 19002 that accesses a Web page's content, a function 
19004 that performs a layout of the web page's content, and functions 19006 and 19014 that 
detects the scale at which the user has selected to have the layout of the web page's contents 
display. 

If the user has selected to have the web page's layout displayed at a given larger display 
scale, function 19006 causes function 19008 to display a portion of the web pages layout at the 
larger scale. This includes perfonning a funcSon 19010 to represent the layout's images with 
bitmap images scaled for display at the larger scale and a function 19012 that represents the 
layout of the web page's strings with bitmaps composed from separate font bitmaps that have 
sizes appropriate for display at the larger scale. 

If, on the other hand, the user has selected a given smaller display scale, one which is so 
small that at least some of the text of the web page cannot be displayed at that scale in a size that 
is readable, function 19014 causes a function 19016 to display a portion of the web page's layout 
at the smaller scale. This includes perfomru'ng a function 19018 that represents the layout's images 
with bitmap images that have been scaled down for display at the smaller scale, and a function 
19020 that represents at lease some strings with bitmaps composed of greefced text 
representations that indicate the size and location of individual strings in the display at the smaller 
scale. 

In many cases the bitmaps used to represents strings in such greeking will merely be lines 
or rectangles having a width and/or height conesponding to the size of their conesponding strings 
in the web page's layout at the small-scale. 

When a layout is displayed at a size in which text Is too small to read the use of greeked 
representations of text can makes such a display easier and more pleasant to see, and such 
greeking generally takes less computation to generate that would corresponding string images 
generated from unreadabily small font bitmaps. 
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One of the major uses the method shown In FIG. 190 Is to enable a user to quickly gain 
an oveiview of a web page's layout and to aitow him or her to quickly select different portions of 
such a web page, sudt as has been described above with regard to FIGS. 136 and 137. 

Those skilled In Ihe art of computer user interfaces will appreciate that some of the 
methods described in FIGS. 185 through 190 can be used In combination with each other and with 
other aspects of the Invention described above as part of a rfngle user interface mode, whereas 
others are them wouW nonnally be used in different user interface or different user interface 
modes. 

It should be understood that the foregoing description and drawings are given merely to 
explain and IBustrale, and that the invention Is not limited thereto except Insofar as the interpretation 
of the appended claims are so limited. Those skilled in the art wrtm have the disclosure before them 
will be abte to make modifications and variafions therein without departing from the scope of the 
Invention. 

The Invention of the present applicafion, as broadly claimed, is not limited to use v\fth any 
one type of operating system, computer hardware, or computer networii, and, thus, other 
embodiments of the invention couW use differing software and hardware systems. 

Furthemwre, it should be understood that the program behaviore described In the claims 
betow, nke virtually all program behaviors, can be perfomied by many different programming and 
date structures, using substentially different organization and sequencing. This is because 
programming is an extremely flexibte art In which a given idea of any complexity, once understood 
by those skilled in the art, can be manifested In a virtually unlimited number of ways. Thus, the 
claims are not meant to be limited to the exart steps and/or sequence of steps described in the 
figures. This Is particuteriy true since the pseud<H5ode described in the text above has been highly 
simplified to tet It more efficiently communicate that whteh one sWiled In the art needs to know to 
implement the inventton without burtlening him or her with unnecessary details. In the Interest of 
such simpiiflcation, the stmcture of the pseudo-code described above often differs significantly 
ftom the structure of the actual code that a skBted programmer would use when implementing the 
invention. Furthemrare, many of the programmed behavtors vi/hlch are shown being perfonned in 
software In the spedfkatlon could be perfonned in hardware In other embodimente. 
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In the many embodiment of the Invention discussed above, various aspects of the 
invention are shown occunring together which could occur separately in other embodiments of 
those aspects of the invention. 

IMost of the various Illustrations of subpixel optimization and non-linear color-balancing 
described in various parts of this specification relate to RGB subpbcel addressable displays having 
vertical subpbcel striping. It should be appreciated that many of inventions innovations relating to 
non-linear color balancing and subpixel optimization can be used with subpbcel displays that have 
BGR or other types of subpbcel addressability, as well as subpixel display having horizontal 
subpixel striping. 

in the non-linear color balancing methods shown above the only portion of a subpixei's 
luminosity distributed by color balancing is that which Is higher than ttie minimum subpbcel 
luminosity value within a pbcel. But in other embodiments other portion of a subpbcei's luminosity 
that cause cotor imbalance within a pbcel could be distributed, such as portions that differ from the 
mean or maximum subpbcel luminosity of pbcel. in such embodiments subpbcel luminosity values 
below such a nfiean or maximum would, In effect, be negative luminosity values, that could be 
dtetributed by a we^hted decreasing of subpbcel luminosities in such a subpbcel's neighbortiood. 

All tiie non-linear color balancing mettiods shown above only distribute ttiose portions of a 
subpbcers luminosity ttiat cause color imbalance within a subpbcel's conesponding pbcel. This Is 
done because tiie anangement of three successive RGB or BGR subpbcels commonly found wlttiin 
a whole pbcel are perceptually well color balanced. If the subpbcels of such a whole pbcels are of 
equal luminosity ttiey tend to appear more color balanced to the eye ttian an isolated set of tiie 
same three colored subpbcels shown at tiie same Intensity In an order in which green is not ttie 
central color. This is one of tiie reason why edges of fonte tiiat appear at otiier tiian whole pixel 
boundaries appear color Imbalanced. 

But ottier non-linear color balancing embodiments need not be limited to only distributing 
subpbcel luminance tiiat causes Imbalance wittiin Individual whole pixels. Ottier non-linear color 
balancing embodiments could detemiine ttie degree of subpbcel color imbalance wittiin regions 
ottier ttian whole pbcels, and distribute subpbcel luminance values based totally or in part on 
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Imbalance with such regions. For example, studies could be performed to find which distributions 
of imbalanced coverage values created a nranimal spatial spreading while rtialntaining the 
perception of color balance, for each of a plurality of commonly occuning imbalance patterns, and 
such perceptually selected distributions could be used to distribute color imbalance that occurs in 
spatial regions other than whole pixel tegbns. 

Certain aspeds of the Invention relate to the creation and use of subpixel optimized 
images that calculate luminosity values for individual pixels by line coverage techniques. It should 
be appreciated that other aspects of the invention claimed below without specific recitation of such 
line or area coverage functions are not limited to such methods of detennlning subpixel luminosity 
and could for example use other known methods for detemnining coverage values with source 
Images comprised of color bitmaps, greyscale bitmaps, fonts, and other shapes, Including, but not 
limited to, area sampling techniques. 

In the discussion above, the source image windows used to assign luminosity or coverage 
values in subpixel-optimized bitmaps are rectangular, and have sfees corresponding a whole pixel 
in a multi-color subpixel-optimized image and conesponding to a subpixel In a bkolor subpixel 
optimized image. In oflier embodiments windows of different shapes and sizes can be used. For 
example, in multi-colored subpixel-optimized images source image windows might have a size 
somewhat smaller that that coresponding to a virfiole output image pixel. In some embo(flments, a 
non-uniform weighting function could be used to translate coverage or luminosity values in a 
source image window Into coverage or luminosity value in the output image. For example, In multi- 
color subpixel-optimized images it might be prefenBd to give more weight to the luminosity In 
portions of a source windows that corresponds in size and location to the subpixel whose 
luminosity is being detentiined. in fact, the line coverage an^ngement discussed above with 
regard to FIGS. 17 through 19 provides such central weighting because its vertical line mns only 
through the portion of the source image window that conespond to the location of the subpixel for 
which Its line coverage values are being detennined. 

Although many aspects of the invention explicitly relate to the use of subpixel optimization, 
many other aspects do not depend on subpixel-optimization. In some such aspects of the 
Invention fomis of anti-aliasing can be used that do not Involve subpixel-optimization. Fonns of 
anfl-aliasing that do not Invoh^e subpixel optimization can allow images to appear to have a higher 
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resolution than could be provided In the absence of such anti-aliasing. This is particularly true for 
font images. For example, fonts as small as seven pixel's per em can be read relatively easily 
provided that they have the right shape, are properly hinted, and use anti-aliasing - either with or 
without subpixel-optimization, although proper subpixel-optimlzation maices such small fonts easier 
to read. 

In this specification and the claims that follow reference to a "screen", particularly a screen 
on which scaled-down images, text, or web page layouts are displayed, can normally include either 
whole screens or parts of screens, such as graphic windows on screens. For example, the scaled 
down screen images refened to might be shown in a window on a considerably larger screen, or 
may be shown on a portion of a small screen ftat is left after space is dedicated to certain 
graphical user Interface elements, such as, for example, the tool bar shown In FIG. 1 14. It should 
also be appreciated that certain subpixel-optimlzed aspects of the invention can be used to display 
Images and/or text across all or a substantial portion of a large screen, such as to allow such a 
large screen to see content at a higher spatial resolution that it could with non-subpixel-optimlzed 
techniques. 

Some aspects of the invention specifically relate to laying out digital content at a virtual 
resolution and then displaying it at a scaled-down resolution. It should be appreciated that in other 
aspects of the invention the images and text of the digital content could be scaled down before 
layout, and then be laid out at the adual resolution they are to be shown at. 
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WHATWECLAIMIS 

-1. A method of displaying digital content on a screen, said method comprising: 
-accessing said digital content including images and text strings; 
-laying out said Images and text at a virtual pixel resolution using layout pixel sizes for said 
images and text, so as to assign a horizontal and vertical virtual position in said layout to 
each of said images and each portion of a string of text displayed on a given line; and 
-drawing at least a portion of said layout on said screen; 
-wherein: 

-the displayed portion of the layout has a displayed pixel resolution that is scaled down by 
a scale factor relative to the pixel resolution of said portion in the layout pertomied at said 
virtual pixel resolution; 

-images and text in said displayed portbn of the layout are shown at pixel coordinates that 
corresponding to the positions of said images and text In the layout, as scaled down by 
said scale factor; 

-the images and text are drawn in said display at scaled-down pixel sizes that correspond 
to the pbcel sizes used for said images and text in the layout, as scaled down by said scale 
factor; 

-the Image of a string of text in said display is composed from a succession of font bitmaps 
having pbcel sizes that are scaled down by said scale factor relative to the size allocated to 
the characters of said string in said layout; and 

-the shape and pbcel alignment of a gh^en character represented in said display by one of 
said font bitmaps have been selected as a function of the given size of said bitmap to 
improve the readability of said bitmap at said given bitmap size. 

-2. A method as in Claim 1 wherein said digital content Is a web page. 

-3. A method as in Qaim 2 wherein said layout image sizes at which images are laid out in said 
web pages are sizes of said images specified by the content of said web page. 

4. A method as in Claim 2 wherein 
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-said method includes allowing a user to select to have said scaled-down display of a 
given layout pertbmned at different scale factors; 

"different font sizes are used in said scaied-down display for similar text at different 
selected scale factors; and 

-the shape and pixel alignment of conresponding characters in the font bitmaps of such 
different font sizes are different to Improve readability of font bitmaps at each of such 
different font sizes. 

-5. A method as In Claim 2 wherein said scaled-down pixel sizes include font sizes of 8 pixels per 
em or less. 

-6. A method as in Claim 5 wherein the characters shapes represented by font bitmaps of said 
pbcel size of eight pixel per em or less have been hinted for improved readability at such size. 

7. A method as in Claim 2 wherein: 

-the size of a given text string at the given scaled-down text size Is a function of the pixel 
dimensions of each character in the string at the given scaled^own pbcel size, where the 
pixel dimensions of each character is detemiined in part at a function of the dimensions of 
' the pixel bitmap needed to represent the particular character's shape at a desired level of 
readability at said scaled-down text size; and 

-the size used for a given string in the layout conesponds to the sb:e of the given string at 
the given scaled-down text size at which said string will be drawn In said scaled-down 
display, scaied-up by said scale factor 

-8. A method as In Claim 2 wherein: 

-said screen is part of a computer having an operating system that displays digital content, 
including text strings composed on said computer from individual font bitmaps, on said 
screen in a portrait orientetion; and 

-said scaied-down display of a portion said layout is drawn on said screen in a landscape 
orientation; 

-wherein said composing of text from Individual font bitmaps composes text in a landscape 
orientation using font bitmaps having a landscape orientation relative to said screen. 
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-9. A method as In Claim 8 wherein: 

-the screen on vMch the scaled-down display is drawn has pixels comprised of a given . 

anBngement of separately-addressable, differently-colored subpixels; 

-said arrangement of subpixels within pixels of the screen cause subpixel color to vary: 

-along a first bitmap display axis relative to bitmaps drawn on said screen in a 

portrait orientation; and 

-along a second, perpendicular bitmap display axis relative to bitmaps drawn on 

said scr^n in a landscape orientation; 
-a given font bitmap used to compose the image of text in said scaled-down display is a 
subpbcel-optimized bitmap that 

-is optimized for display in which said subpixel color variation occurs along said 

second display axis relative to said font bitmap; and 

-assigns a luminosity value to each given sub-pixel of a screen pixel having said 

given anrangement of differentiy-colored subpixels that is drawn to by said font 

bitmaps as a function of: 

-a coverage value representing the percent of the given subpixel that Is covered 

by a character shape being represented by the font bitmap; 

-In the case of at least some subpbcels of said font bitmap, a cotor balancing 

distribution of a percent of the given subpixel's coverage value from said coverage 

value to coverage values of nearby subpixels, including subpixels of different 

colors, made to a prevent color imbalance that would result from the difierence 

between the gNen subpixeFs coverage value and the coverage values of a given 

set of one or more nearby subpbcels of different colors; and 

-In the case of at least some subpixels of said font bitmap, such a color balancing 

distribution to the given subpixel's coverage value of a portion of coverage values 

fmm one or more nearby subpixels. 

-10. A method as in Claim 2 wherein different portions of text specified by the web page as being 
different types of text that are commonly displayed with different size fonts are represent with fonts 
of the same size before perfomiing said layout and display to allow a greater quantity of readable 
text to fit in the scaled-down display. 

-1 1. A method as in Claim 1 wherein: 
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•the font bitmaps used to compose the Image of text in said scaled-down display are antl- 
allased bitmaps that assign a color value to a given screen pixel as a graded function of a 
coverage value representing the percent of the given pixel that is covered by a character 
shape being represented by the font bitmap; and 

-the shape and pbcei alignment of a character represented by such a font bitmap has been 
selected to Increase the degree of alignment of edges of the character shape with pixel 
boundaries of the font bitmap as a function of the particular pixel size of each such a font 
bitmap. 

-12. A method as In Claim 11 wherein: 

-the screen on which the scaled-down display is drawn has pixels comprised of a given 
arrangement of separately-addressable, differently-colored subpixels; 
-the anfi-allased font bitmaps used to compose the Image of text in said scaled-down 
display are subpixel-optimized bitmaps that assign a luminosity value to each given 
subplxel of a screen pb(el having said given anangement of differently-colored subpixels 
as a function of: 

-a coverage value representing the percent of the given subplxel that is covered 

by a character shape being represented by the font bitmap; 

-in the case of at least some subpixels of said font bitmaps, a color balancing 

distribution of a percent of the given subpixeFs coverage value from said coverage 

value to coverage values of neart3y subpixels, including subpbcels of different 

color, made to a prevent color imbalance that would result from the difference 

between the given subpixel's coverage value and the coverage values of a given 

set of one or nrrore nearby subpixels of different colors; and 

-in the case of at least some subpbcels of said font bitmaps, such a color 

balancing distribution to the given .subpbcel's coverage value of a portion of 

coverage values from one or more nearby subpixels. 

-13. A method as in Claim 12 wherein said color balancing distributions only distribute portions of a 
subpbcel's coverage value that causes color Imbalance within the vifhole pbcel of which it is part 



-14. A method as in Claim 12 wherein: 
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-the images drawn In said scaled-down display are subplxel-optimized images that assign 
a luminosity to each differently-colored subpixel In the display of such a scaled-down 
Image as a function of the amount of luminosity of the given subpixel's color found in a 
window associated with the given subpixel in the higher resolution image the scaled-down 
image represents; and 

-the window In the higher resolution image associated wift each subpixel of a given pixel 
has a different position relative to the higher resolution image that conesponds to the 
different position of Its conesponding subpixel In the scaled-down image. 

-15. A method as in Clsrim 1 wherein: 

-a first computer device performs said accessing of the digital content, laying out of said 
digital content; said scaGng down of said images; 

-a second computer device has said screen and perfomis said drawing of the scaled-down 
display on said screen, including the composing of text images from font bitmaps; and 
-said cooitJinates of Images and text produced as a result of said layout, said scaled down 
images, and said text contained in said digital content are download from said first 
computer device to said second computer device. 

-16. A method as in Claim 1 wherein: 

-said digital content Is a screen image produced at sad virtual resolution by one or more 
application programs; and 

-said scaled-down display shows said portion of said screen image vwth the Images and 
text of said screen image and their positions scaled down by said scale factor. 

-17. A method of displaying a web page comprising: 

-accessing the web page, including one or more images and one or more text strings; 

-displaying in a landscape orientation, in a scaled-down manner, a portion of said web 

page, including at least some of Images and text strings; 

-wherein said displaying is perfonned on a screen of a computer having an operating 

system that displays an associated graphical user Interface on said screen in a portrBit 

orientation; 

-wherein the scaled-down displaying of said web page includes: 
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-displaylng a given images at a scaled-down pixel size; and 
-dispiaying a given text strings witli a string image composed on said computer 
from a plurality of font bitmaps corresponding to the characters of said string when 
displayed in said landscape orientation; and 

-the shape and pixel alignment of a given character represented in said display by 
one of said font bitmaps have been selected as a function of the given size of said 
bitmap to improve the readability of said bitmap at said given bitmap size. 

18. A method as in Claim 17 wherein the operating system can only display said graphical using 
internee on said screen in said portrait orientation. 

19. A method as in Claim 17 wherein the operating system can display said graphical using 
Interfeice on said screen in either said portrait or said landscape orientation. 

-20. A method as in Claim 17 wherein 

^atd method includes allowing a user to select to have said scaled-down display 

perfbnned for a gh^en web page at different scale Actors; 

-the scale-down pbcel size at which each images is shown In displays perfonned at 

different scale factors varies as a function of said different scale Actors; 

-the sizes of the font bitmaps shown in said string Images in displays perfomned at different 

scale factors varies as a function of said different scale factors; and 

-the shape and pixel alignment of a given character in the different font bitmaps shown for 

the given character in displays perfonned at different scale factors are difterent to improve 

readability of the character at each of the different font bitmap sizes used to represent the 

character at different scale factors. 

-21. A method as in Claim 17 wherein: 

-the font bitmaps used to compose one or more of said string images are anti-aliased 
bitmaps that assign a color value to given screen pbcel as a graded function of a coverage 
value representing the percent of the given pixel that is covered by a character shape 
being represented by the font bitmap; and 

-the shape and pixel alignment of a character represented by such a font bitmap has been 
selected to increase the degree of alignment of edges of the character shape with pixel 
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boundaries of the font bitmap as a function of the parHcular pbcel size of each such a font 
bitmap. 

-22. A method as in Claim 21 viAierein: 

-font bitmaps used to compose string Images include small font bitmaps having a small 
font size of eight pixels per em or less; and 

-the shape and pbcel alignment of a character represented by such a small font bitmap has 
been selected to Increase the degree of alignment of edges of the character shape with 
pbcel boundaries of the small font bitmap as a luncfion of the particular pbcel size of each 
such small font bitmap. 

-23. A method as In Claim 22 wherein the font bitmaps of said small font size represent a majority 
of characters of the Roman alphabet within an advance width of four pbcel columns or less. 

-24. A method as In Claim 23 wherein the font bitmaps of said small font size represent a majority 
of lowercase letters with an x-helght greater than four pixels. 

-25. A method as In Claim 21 wherein: 

-the scrsen on which the scaled-down display is drawn has pixels comprised of separately- 
addressable. differentlyKX>fored subpbcels, in which the differently colored subpbcels of 
each pbcel have a first subpbcel anangement when said screen viewed in the landscape 
orientation, and a second subpixel anangement when the screen viewed In the portrait 
orientafion; 

-the anti-aliased font bitmaps used to compose said text Images are subpbcel-optimized 
bitmaps that assign a luminosity value to each given subpbcel of a screen pixel having said 
first subpixel anangement as a function of: 

-a coverage value representing the percent of the given subpbcel that is covered 
by a character shape being represented by the font bitmap; 
-in the case of at least some subpixels of said font bitmaps, a color balancing 
distribution of a percent of the given subpixeFs coverage value firom said coverage 
value to coverage values of neart)y subpixels. including subpfacels of different 
color, made to a prevent color imbalance that would result from the difference 
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between the given subplxel's coverage value and the coverage values of a given 
set of one or more nearby subpixels of different colors; and 
-in the case of at least some subpbcels of said font bitmaps, such a color 
balancing distribution to the given subpbcei's coverage value of a portion of 
coverage values from one or more neari}y subpixels. 

-26. A method as In Claim 25 wherein: 

-the subpbcels of a given pixel in said screen vary in color along the horizontal direction 
when said screen is viewed in said landscape direction; 

-so the added resolution made possible by subplxel optimization increases the horizontal 
resolution available to show text characters drawn on said screen in the landscape 
orientation. 

-27. A method as in Claim 25 wherein said color balancing distributions only distribute portions of a 
subpbcel's coverage value that causes color imbalance within the whole pixel of which it is pari 

-28. A method as in Claim 27 wherein the character shapes represented by said subpixel- 
optimized font bitmaps and the alignment of such shapes to the pixels in said bitmaps have been 
selected as a function of the size of such bitmaps to improve the alignment of the edges of such 
shapes which edges of bitmap pixels, so as to decrease the differences between subpbcei 
coverage values within the pixels of such bitmaps that require color balancing to prevent color 
imbalances. 

-29. A method as In Claim 17 wherein said computer is a handheld computer. 

-30. A method as in Claim 17 wherein: 

-the computer on which said scaled-down display is shown requests a web page from a 

remote computer over a computer network; 

-said remote computer accesses said web page; 

•said remote computer lays said web page outto detentilne relative positions 

con^sponding to the relative positions at which said images, strings, and linics are to be 

displayed; and 
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-said remote computer downloads said Images, strings, links, and relative positions over 
said computer network to said display computer, and 

-said display computer draws said Images, strings, and links at relative positions on said 
screen determined as a function of said downloaded relative positions. 

-31. A method as in Claim 30 wlierein said remote computer 

-scales down each of said displayed images to saW scaled-down pixel size and downloads 
said scaled-down image to the display computen 

-perfonns said layout based on a font metrics detennined for each strings as a function of 
the size of the indhridual font bitmaps that will be used to compose the string image said 
display computer; and. 

32. A computer system comprising: 

-one or more computer readable memory including program instructions for perfomiing the 
functions of the method of any one of the claims numbered 1 through 31; and 
-one or more processors for executing said instmctions. 

33. A one or more computer readable memories including program instructions for performing the 
functions of the method of any one of the claims numbered 1 through 31 . 
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-subpixel optimization routine (using line coverage)"-2900 
-for eacli row In output image-^2901 
-for each pixel in row-2902 

-for each subpixel in pixeh2904 

-for each of its scan lines-'2906 

-calculate each Intersection between that scan line 
and a pixel boundary~2908 
-for each portion of a scan line which occurs 
between two scan line ends, a scan line end and a 
pixel boundary, or two pixel boundaries~2910 

-add to a coverage value associated with the 
subpixel the multiple of the percent of that 
scan line covered by the portion times the 
component color value of the pixel covering 
that portion corresponding to the color of the 
cun-ent subpixel, all divided by the number 
of the subpixel's scan lines'-2912 
-Set the pixel's color value equal to a color having an compound 
RGB value with red, green, and blue values equal to the subpixel 
luminosity value of the pixel's red, green, and blue subpixels, 
respectively.-2914 



FIG. 29 
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-subpixel optimization routine (using area coverage)^! 00 
-for each pixel row In output lmage'-4102 
-foreacli pixel in row-4104 

-for each subpixel In a pixel'-^ioe 

-determine which pixels of source image are in a source 
image window associated with the subpixeM108 
-for each such included source image pixeMHO 
-calculate the percent of the subpixel's source 
image window's area covered by the pixel's 
area-4112 

-add to a luminosity value being calculated for the 
subpixel the multiple of the percentage of the 
subpixePs source image window area covered by 
the pixel times the pixel's color component value 
corresponding to the color of the cunrent subpixel 
^114 

-Set the pixel's color value equal to a color having an compound 
• RGB value with red, green, and blue values equal to the subpbcel 
luminosity value calculated for the pixel's red, green, and blue - 
subplxels, respectively.~41 1 6 



FIG. 41 
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FIG. 48 



wo 02/088979 



PCT/US02/14380 



21/87 



4740 4742 4744 4746 4748 4750 4752 4754 4756 4758 4760 4762 



\ \ \ \ \ 



I M I / 




5000 



M W .M . I M 



5002 




I 



W M M I M M 



5004 




4910 



FIG. 49 



wo 02/088979 



22ffi7 



PCT/US02/14380 




FIG. 50 




wo 02/088979 PCT/US02/14380 

23^7 



-subpixel optimization for bicolor bitmap -5300 
-for each pixel row in image-5301 

-for each pixel in row-5302 

-for each subpixel in the pixeh5304 

-determine which pixels of source image are in a window 
portion of source Image corresponding to subpixel's area 
in scaled image-5306 

-for each source image pixel all or partially in the 
subpixel's source Image window-'5308 

-calculate the percent of the window's area covered 

by the pixel's area-SSI 0 

-add to a luminosity/coverage value calculated for 
the subpixel the multiple of the percentage of the 
window area covered by the pixel times the pbcel 
average foreground color intensity-5312 
-find the minimum subpixel luminosity/coverage value so 
calculated for the pixeI-5314 
-for each subpixel in pixeh531 6 

-set a iuminosity/alpha value being calculated for the sub 
and the pixel to the pixel's minimum subpixel 
luminosity/coverage value-53 1 8 
-distribute that portion of the subpbcel's 
luminosity/coverage value that exceeds the pixel's 
mininum subpixel luminosity/coverage value to the 
luminosity/alpha values being calculated for the subpixel 
and adjacent subpixels in the pixel row using a color 
balance distribution filter-5320 
-For each pixel in row-5322 

-set the pixel's color value equal to a color having an compound 
RGB value with red, green, and blue component values equal to 
thise luminosity/alpha values calculated for the pixel's red, 
green, and blue subpixels, respectively.-5324 
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-subpixel optimized font bitmaps with non-linear color balance'-6000 
-for each pixel row-6002 

-for each subpixel in a row-6004 

-determine a coverage value representing the perecent the 
subpixel which Is covered by the font shape-^SOOB 
-for each pixel in row-6008 

-determine the minimum coverage value calculated for each of 
its three subpbcels -6010 

-add minimum to temporary alpha value being calculated for 
each subpixel of the pixeh601 2 
-for each of pixel's subpixels-6014 

-determine excess of subpixel's coverage value over the 

pixel's mlnimum--6016 

-distribute this excess value into sub-pixel alpha values 
based on color balance distribution filter appropriate for 
sub-pixel's color-^SOIS 
-for each pixel in row-6020 

-use the three color alpha value defined for each pixel by the 
three alpha values calculated for its subpixels In a look-up table 
to map that value into one of relatively small palette of 
colors'-6022 

FIG. 60 
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-Center-Weighted Color Balance Filter-9400 

-For Coverage Value 0 0,0,0,0,0 

-For Coverage Value 31 0,1 ,1 ,1 ,0 

-For Coverage Value 62 0,2,2,2,0 

-For Coverage Value 93 1 ,2,3,2,1 

-For Coverage Value 126 1 .3,4,3,1 

FIG. 94 



-Asymetric Color Balance Filter-9500 

-For Coverage Value 0 0,0,0,0,0 

-For Coverage Value 31 1,1,1 ,0,0 

-For Coverage Value 62 2.2,2,0,0 

-For Coverage Value 93 3,3,3,0,0 

-For Coverage Value 126 3.3,4,1 ,1 
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-Creation of input-color-to-output-color looI<-up table^-QBOO 

-run the ciiaracters of multiple fonts through non-linear algorithm for deriving 
subpixel optimized font bitmaps, while keeping a histogram of the number of 
times each of one of the 2196 possible three color alpha values is calculated 
forapixeh9602 

-create a 122 color output palette by-9604 

-selecting the thirteen grayscale colors possible for whole pixel alpha 
values in which each subpixel can have one of thirteen levels including, 
black and white-^Seoe 

-selecting the 109 other most frequently occuning colors in the 
histogram'-9608 
-for each of the 2196 possible calculated alpha values-9610 

-if that input color exactly matches one of the pallette's colors-9612 

-associate the Input color with that palette, or output, color-9614 
.ifnot-9616 

-for each of 122 output colors'-9618 

-if (ri-ro) and (gi-go) are of same sign and if 
|ro.go| < |ri-gi|+x~9620 

-calculate the distance from the Input color to the 
output color-9622 

-if that distance is the closest distance so far to the 
input color'-9624 

-save it as closestAllowedPaletteColor-9626 

-associate the input color with the 
closestAllowedPaletteColor'-9628 
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-Displaying text with font bitmaps having subpixel alpha value components -9700 
-for each string to be displayed-9702 

-sample points In rectangle of bitmap in which it is to be drawn to 
determine its background color'-9704 

-for each of the 122 whole pixel alpha values used to represent font 

bitmaps-9706 

-for each of the three subpixel component colors'-9708 

-calculate color luminosity for the subpixel component 
color as the cun^ent subpixel's alpha value in the current 
whole pixel alpha value ( the current supbcePs alpha) 
times luminosity of the cun-ent subpixel color in the 
foreground color with which the fonts are to be drawn plus 
(1 minus the current subpixel's alpha) times the luminosity 
of the current subpbcel's color in the background 
color-9710 

-map the current whole pixel alpha value to the whole pixel color 

comprised of the three subpixel luminosities calculated for the 

whole pixel alpha value^9712 
-for each character of string to be displayed-9714 

-access its associated font bitmap'-9716 

-for each of pixel of the font bitmap--9718 

-find the color value which has been mapped to the pixel's 
corresponding whole pixel alpha value In the font 
bltmap~9720 

-set the con*esponding pixel In the subpixel addressable 
display to that whole pixel color value--9722 
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HTML 



-Download Display List 
-dearCmd 
-scroilCmd 

-badcgroundColorCmd 
HBdanglleCmd ^ 
-imageLocationCmd 
-FontCmd 

-StringCmd ^ 
-ContaolCmd 
-4mageCmd 
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-if receive download 
-redangleCmd 
-strihgCmd 
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-Sample html~10300 
-</heacl> 
-<body 

background="http://a1636.g.akamai.net/7/1636/797/e5e77dd148cc98/graphics 
.boston.com/gIobe/images/tiles/tile.gir linl<="6C331 7" vlinl<="5C4033" 
onLoad="FrameTliis()'*> 



-<P> /I 0300 

^ /10300 

-<B>E-mail to a frlend</b><BR> ^ 

-<a href="http://tools.boston.com/pass-it-on/popularr>See wliat stories users 

are sending to friends</a><p> 

^^^10300 ^10300 
-<B>Free headlines e-mail</B><br> jr 
-<a href="/newsletlers/headlinesr>The best of the Globe each weekday 
moming<a><p> 

^10300 

-<B>Alternative vi^</b><BR> ^ 

-<a hre^=7globe/lowgraphics">Low-graphics version<a><br> 

-<a href=7g[obe/acrobat/today.pdf >How it looks in print</a> 

-<P> / 10302 ^ 

/ 10300 
-<l-S^TlONS-> 
.<IMG WIDTH="120" VEXQWK^IT 

SRC="http://a1636.g.akamai.net/7/1636/797/0df5e88d0bb528/graphics.boston 
.com/globe/images/navs/nsections.glf border="0" alt="Sections" VSPACE="r 
WIDTH="120" HE1GHT="27"><BR> 
-<a href=7globe" onMouseOver="loadimageCpageone', 
'http://graphics.boston.com/globe/images/navs/rpageone.gif); status- Boston 
Globe Online: Page One'; return tme;" onMouseOut="loadimageCpageone', 
'httpy/graphics.boston.com/globe/images/navs/npageone.glf);status=";retum 
tme;"><IMG WIDTH="119" HEIGHT="14'' 

SRC="http://a1636.g.akamai.net/7/1636/797/f0a63d528dc7e3/graphics.boston 
.com/globe/images/navs/npageone.giT border="0" alt="Boston Globe Online: 
Page One" name="pageone" VSPACE="1"></a><br> 
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-Browser's Proxy Code-'10500 

-if receive request from thin client for a web page-10502 

-relay request to server indicated in URL of request-10504 
-if receive indication tliat browser has completed a screen draw or 
redraw-IOSOe 

-call the screen capture and download routine when screen redraw is 

complete -10510 
-if receive control object state query from browser'-10514 

-query thin client for state of indicated one or more controls-^IOSIS 

-send those control states to browser-10518 
-if receive scroll/move command from thin client--'10520 

-move view window accordingly relative to browser's virtual 

screen~10522 

-if portion of view window which was in view window before move Is still 
in view window-'l 0526 

-place appropriate scroll command at start of download display 

list-10528 

-if moved view window includes portion of web page not currently In 

virtual screen'-10530 

-scroll browser's virtual screen accordingly'-10532 
-request redraw for the newly scrolled virtual screen from 
browser'-10534 
-if receive zoom command from thin client-10536 

-change view scale factor accordingly-^IOSSS 

-scale view window accordingly relative to browser's virtual 

screen-10540 

-if scaled view window Includes portion of web page not cun-ently in 
virtual screen-10542 

-scroll virtual screen change its resolution to cause scaled view 

window to fit in virtual screen--^ 0544 
-call for screen redraw'-10552 
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-if receive virtual resolution command from thin client'-10554 

-change browser's virtual screen resolution to requested virtual 

resolution~10556 

-call for screen redraw-IOSBO 

-if receive other user input event from thin client-'10562 

-transfonm its screenXY on client screen to corresponding location in 
browser screen using location of view window and display scaling 
factor-10564 

-relay event to browser's event queue-10566 
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-Screen Capture And Download Routine^-IOBOO 
-ask for browser for screen redraw-10602 
-if browser calls-^IOStM 

-measureString-1 0606 

-map requested font family and font size into substitute font 
family and size, including:- 10608 

-select size for substitute fonts as a function of the 
requested font size and downscaling by the display 
scaling factor'-'' 0610 

-replace smaller size fonts with nanrower and taller size to 
take advantage of the higher ratio of horizontal resolution 
of subpixels~10612 
-if limitMinimumFontSize Is on~10614 

-prevent substitute font size from being below a 
minimum pixel size^-ioeie 
-retum string measurement for substituted font and font size, 
scaled up measurements by display scale factor--10618 
-stringDraw-1 0620 

-transfonn string's virtual screen screenXY to thin client 
screenXY by scaling and/or translating as a function of the 
display scale factor and the view window's position relative to 
virtual screen-10621 

-If substituted font family and size associated with string in prior 
measure string call and any other font attributes for string is 
different than cunrent font attribute for end of display list~10622 
-store a font command at end of display list changing 
cunrent font attributes to attributes corresponding to 
current string'-10623 
-store string, Its thin-cllent screenXY, and Its substituted font, 
including size and color, at end of download display llst'-10624 
-rectangleDraw^-'l 0626 

-transform rectangle's virtual screen screenXY, width, and height 
to thin client screenXY, width and height by scaling and/or 
translating as a function of the display scale factor and the view 
window's position relative to virtual screen-1 0628 
-if rectangle's color Is different than background color for current 
end of download display list-10630 

-add a background color command changing to new 
background color at end of download display list-10632 
-store rectangle, transformed screenXY, width, and height at end 
of download display list-'10634 
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-bitmapDraw-10636 

-If images URL Is not already in a download image list-1 0638 
-if bitmap is color bitmap-10642 

-scan image for one or more portions of sufficient 
size whicii have only colors from a given bicolor 
spectmm-lOB^^ - 

-for each bicolor portion of image found^-l 0646 
-perform bicolor subpixel optimization, 
scaled down by display scale factor, on 
portion using opposite ends of its bicolor 
spectrum as foreground and background 
color at display scale factor~10648 
-if foreground color is too chromatically 
unbalanced, render Image with a substituted 
more-balanced foreground color-10650 
-for each non-bicolor portion of image found'-IOeSZ 
-perfomi multicolor subpixel optimization, 
scaled down by display scale factor, on 
bitmap using color image subpixel algorithm 
at display scale factor-1 0654 
-else if bitmap is grayscale bitmap--10656 

-perform bicolor subpixel optimization, scaled down 
by display scale factor, on bitmap using black and 
white as foreground and background colors at 
display scale factor--1 0658 
-store scaled-down, subpixel-optimized bitmap, with a 
unique imagelD, transfonned width and height, and its 
URL at end of image list-10662 
-transform image's screenXY for download and store an image 
location command having the imagelD and the transformed 
screenXY, width, and height stored for the image in the image 
list at end of download display list-10664 
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-controlCreate-10666 

-transform contrors screenXY by scaling and/or translating as a 

function of the display scale factor and the view window's 

position relative to virtual screen-IOSBZ 

-place corresponding control command, its transformed 

screenXY, and corresponding text in download list-10668 

-create corresponding browser-side portion of distributed 

controh10670 

-when screen redraw is complete~10672 

-call download display list routine-10674 
-clear display list-10676 

FIG. 106C 



-download display list routine-^l 0700 

-select all elements in display list which will be all or partially be included in the 
new image which is to be created on the thin client's display screen in a 
download stream~10702 

-place all bitmaps in image list corresponding to one or more image locations 
commands in download stream at end of download stream, perfomiing lossy 
compression on them first-'10704 

-user a lossless compression algorithm to compress download stream--10705 
-open a the socket connection between browser's computer and thin 
client--10706 

-send display list to thin client over socket connection-10708 

FIG. 107 
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-Download Stream~10800 

-clearCmd~10802 
-scrollCmd + XYShlft-10804 

-backgroundColorCmd + color~ 10806 
-rectanglleCmd +ScreenXY + width + Height~10808 
-rectanglleCmd +ScreenXY + width + Helght~10808 

-backgroundColorCmd + color~10806 

-ImageLocationCmd I^ ImagelD + ScreenXY+ width + height~10810 

-FontCmd + FontAttributel + NewValuel + FontAttribute2 + 

NewVa!ue2~10812... 

-StringCmd + ScreenXY + String~10814 

-StringCmd + ScreenXY + String-10814 

-imageLocattonCmd + ImagelD + ScreenXY+ width + helght~10810 
-FontCmd + FontAttribute3+ NewValue3...-10812 
-StringCmd + ScreenXY + String~10814 

-StringCmd + ScreenXY + String~10814 

-StringCmd + ScreenXY + String~10814 

-rectanglleCmd +ScrBenXY + width + Height-10808 

-ControlCmd + ControllD + ControlType + ScreenXY + ControlTextUst~10816 

-StringCmd + ScreenXY + String~10814 

-ControlCmd + ControllD + ScreenXY + Control Label-10816 

-ImageCmd + ImagelD + width + height + Bitmap~10818 
-ImageCmd + ImagelD + width + height + Bitmap~10818 
-ImageCmd + ImagelD + width + height + Bitmap-'10818 
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-thin client code~10900 

-If receive a download stream, start responding to individual commands in 
stream in the order in which they are received as soon as one or more are 
received, including responding to each of the following commands as 
follows-10902 

-clearCmd~10904 

-clear thin client screen-^IOOOe 
-scrollCmd-10908 

-bitblit portion of screen which remains on screen after XYShift to 
appropriate position after that shift-^IOQIO 
-clear rest of screen-10912 
-backgroundColorCmd--1 0914 

-set current rectangle background color to the color specified in 
command'-10916 
-rectangleCmd~10918 

-draw rectangle with upper left hand comer at ScreenXY, having 
the width and height specified in command, using cunrent 
background color-10920 
-imageLocationCmd- 1 0922 
-do nothing-10923 

-FontCmd + FontAttributerl + NewValuel + FontAttribute2 + 
NewValue2...-'10924 

-set cun-ent value of all font attributes listed in command to the 
corresponding values listed in the command-10926 
-StringCmd + ScreenXY + String-10928 

-if thin client does not have font bitmap for each character of the 
specified string at the cun-ent font size and font family-10930 
-send separate HTTP request for font of each such 
bitmaps from font server, specifying size, character, font, 
and that is to be subpixel optimized, and subpixei array 
type~10932 

-when receive each requested font-10934 
-place it in font bitmap cache'-'10936 
-if have all characters of string specified in the command--10938 
-draw string, using cun-ent font attributes values, including 
foreground color, and using color from portion of screen 
on which it is being written as the background 
color^10940 
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-ControlCmd + Contrail D + ControlType + ScreenXY + 
ControlTextLjst'-10942 

-If no control has been created having controllD specified In 
command-10944 

-create thin client side of distributed control associated 

with that controllD-10946 
-draw specified control type on screen using subptxel optimized 
bitmaps for control image, at specified screenXY, using drawing 
one or more text items In controlTextList using subpixel 
optimized text, and set control's associated screen hotzone, If 
any to the appropriate portion of screen-l 0948 
-lmageCmd-1p950 

-for each imageLocationCmds in display list having same 
lmagelD-10952 

-draw bitmap at that location'-10954 
-redraw all other items in display list which occur at the same 
location as any of these drawn bitmaps-1 0956 

-if user clicks hotzone assocated with text ent^ field~10958 
-execute keyboard mode routine'-10960 

-display pop up user keyboard and text edit line, saving the 
bitmap which was in its location--10962 
-until user presses enteral 0964 

-if user user types text character-^IOgse 

-place corresponding subpixel-optlmized text shape 
on text edit line--10968 

-add character to position corresponding to cursor 
in a temporary text-edit string variable-1 0970 

-when user press enter, -10972 

-store text-edit string state in corresponding text field 
controh10974 

-draw text-edit string in bitmap of text entry field using 
subpixel optimized fontS'-'l0976 
-remove popup keyboard, replacing the bitmap with was 
In Its place before it was displayed--10978 
-else if user clicks on hot zone of a button or menu item control'-10980 
-change appearance of button or menu item appropriately-10981 
-send event along with button or menu item ID up to proxy~1 0982 
-else if user clicks on hot zone of another type of thin client controh10983 
-change appearance of control appropriately-10984 
-store user selected state change~10985 
-else if user clicks on other portion of screen not associated with the thin client 
program's or its computer's control interface--10986 

-send even with screen location up to proxy-10987 
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-if receive query from proxy process re state of control-^ 0988 

-query state of corresponding control on thin client-^IOQSQ 

-transmit state to browser-10990 
-if user enters a command to scroll screen^-IOQQI 

-upload command to proxy'-10992 
-if user enters a command to change zoomr 10993 

-upload command to proxy-10994 
-if user enters a command to change virtual resolution,'-10995 

-upload command to proxy-10996 
-if user enters another command associated with thin client's control 
GUh10997 

-...-10998 
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-proxy browser code with use of page layout caching-'l 1 500 

-if receive request for web page with an associated view setting-^l 1 502 
-request web page from server-1 1 504 

-when a web page is received from a server--^ 1506 

-have layout engine layout web page at the virtual screen resolution 
specified in the view setting, substituting fonts for layout engine*s 
measure string calls according to view setting's appropriate scale 
factor-11507 

-select virtual screen position relative to resulting layout which will fit the 
view window implicit in the view setting, and redraw screen-1 1508 
-when receive image referenced in web page-1 1518 

-scale and subpixel optimize image according to scale 

factor-11520 

-once have all images referenced In web page--1 1522 
-compress layout and images'-l 1 523 
-download layout followed by images--11524 
-if receive request from thin client to rescale and subpixel optimize bitmaps of 
images at a given scale^-l 1526 

-rescale and subpixel optimize them-1 1528 
-compress them--11530 
-download them to thin client-l 1532 
-if input event from thin client-l 1 534 

-if its layout coordinate associated with it is not currently on virtual 
screen~11536 

-scroll virtual screen so that it is-'l 1538 
-calculate virtual screen coordinate corresponding to layout 
coordlnate-11540 

-place input event with calculated virtual screen coordinate in browser's 
event queue -11542 

FIG. 115 
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-thin client code with use of page layout caching-1 1600 

-if start receiving downloaded page layout display list-^l 1 602 

-set mapping of view window to page layout and calculate scale factor 
as function of view selection--1 1604 

-display received elements of display list which fall within view window 
at current mapping of screen window to display list, including current 
scalefactor-1'1606 

-if user generates input to changes size or location of view window relative to 

downloaded layout-11616 

-make the corresponding change to the mapping of view window to 
display list and calculate scale factor as function any such 
changes--11618 

-display any portions of page layout which fall within current view 
window at current scale factor including-^l 1620 

-displaying strings with font sizes which are function of current 
scale factor, and adjusting for disproportionate changes in size of 
individual or all characters as font sizes change, by changing 
spacing between characters'-l 1622 
-if there is a new scale factor-1 1624 

-requesting proxy to re-scale at new scale factor and 
subpixel optimize all images on screen at new 
mapping-11626 

-scaling at new scale factor and re-subpixel optimizing, 
and displaying the previously downloaded bitmaps of all 
on-screen lmages'-11628 

-when newly scaled images are received, displaying them 
In place of locally rescaled versions*-! 1630 
-if user generates input event relative to screen~1 1 632 

-use view-window-to-page-layout mapping to detemnine location of 

input in layout cooridinates-^'i 1634 

-send Input event and page layout coordinates to proxy^l 1636 
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-Client code for selected text reflow-12900 

-If user selects area of downloaded page layout for text reflow at a new scale 
factor-12902 

-select all strings and corresponding underlines in layout which are substantially 
within selected layout area-'ISSCM 

-label any group of one or more strings whose closeness in layout 
Indicates that they are part of the same paragraph-12906 
-reflow and display the text of each group of strings labeled as a 
paragraph across screen area's across boundaries at new scale factor, 
underiining text In reflowed lay that was underiined before~12908 
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-font server code-1 3900 

-if receive http request for one or more cliaracters of a font-'l 3902 

-if there Is a font file matching having path name spacified in http 
request--13904 

-send that file over network In an http response to network 
address from which the font request came~13906 
-charge account associated with transactlon--13908 
-else if font request is for a font bitmap-13910 

-generate font bitmap file having attributes indicated for font by 
path name, including-13912 

-if font request speclfiestaht a subpixel optimazed version 
ofthefontlsdesired-13914 

-generates subpixel optimized font of character 
using non-linear color balancing~13916 

-send that file over network In an http response to the requesting 
address^ "13918 

-cache the font bitmap file at address corresponding to path 

name specified in request^ "1 3920 

-charge account associated with transaction'-'l3922 
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-rectangIleCmcl~14500 

-draw rectangle with screem position, height, and width defined with higher 
resolution than screen pixel resolution, using bicolor subpixel optimization and 
using current background color-14502 



FIG. 145 



-downloading web applets which creates subpixelized elements on screen-14600 
-server and/or proxy--14602 

-In response to request for media from thin client download media 
Including applets-14608 

.client-14604 

-request medla-14606 

-receive media including applet-14610 

-load and run applet-14612 

-applets draw subpixel optimized elements to subpixel addressable 
screen on client-'l4614 



FIG, 146 



-subpixel optimization of 3-d animations'-14900 
-for each of successive frame times-14902 

-run 3-d animation engine to create bitmap of curent frame, or at least 
of those portions of image which have changed since the last frame, at 
a higher resolution than the resolution at which subpixel-optimized 
Images will be dlsplayed-14904 

-scale-down and subpixel optimize frame bitmap, or at least of changes 
in it since the last frame- 14906 

-display scaled-down subpixel optimized frame bitmap, or at least 
scaled subpixel optimized bitmaps of changed portion of frame-14908 
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-game server computer-15000 

-if have receive user input from one or more game client computers"-15002 
-feed it to game engine-15004 

-have game engine compute display list for current frame (or changes to 
display list for cun-ent frame)~15006 

-have 3-d rendering routine render bitmap frame of current display Iist{or 
current changes to display list) at higher resolution than that at which 
corresponding subpixel-optlmized bitmaps will be shown~15008 
-scale-down and subpixel optimizie current frame bitmapfor bitmaps of current 
changes to frame and their scaled down Iocationsh15010 

-compress one or more successive scaled-down subpixel optimized bitmaps 
(and their locations) -15012 

-download compressed, scaled, subpixel-optimized animation frames (or 
changes and their locations) to game client'-15014 

FIG. 150 



-game client-'15100 

-receive downloaded images (and screen locations)--15101 

-decompress animated images (and screen locations)~15102 

-displays the scaled, subpixel optimized animation frame bitmaps (or change 

bitmaps at their respective positions)--15104 

-if have received any user input^l 5i 06 

-upload user input to game server-1 51 08 
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-subplxel optimization of images with transparency maps'-isaoo 

-produce scaled, either a bicolor or multicolor subplxel-optimized bitmap of the 
foreground image-'15202 

-produce a con-espondingly scaled, bicolor subplxel optimized bitmap of the 
images transparency map^15204 

-display foreground image's bitmap on a subpixei optimized display 

lncluding:-15206 

-for each pixel row of the displayed image--1 5208 
-for each subpixei of such row-15210 

-set currentAlpha to the alpha value of the corresponding 
subpixei of the transparency map-^l 5212 
-set the luminosity of the cun-ent subpixei to currentAlpha 
times the luminosity of the corresponding subpixei of the 
foreground image plus (1 - currentAlpha) times the prior 
luminosity value of the cun^ent subpixeHl 5214 

FIG. 152 



-subpixei optimizing video having interpolation between keyframes-15300 
-decompress video~15302 
-scale and subpixei optimizing key frames-^ 53040 
-scale but do not subpixei optimize Interpolated changes between keyframes 
because of its rapid speed--15306 

-display scaled video on subpixei addressable display with subpixei ootimized 
keyframes and non-subpixel optimized interframe interpolation-15308 

FIG. 153 



-subpixei optimizing video representing clianges to portions of frame-15400 
-decompress video~15402 
-scale and subpixei optimizing frames-'' 5404 

-scale and subpixei optimize change bitmaps, scale their location relative to 
frame-15406 

-repeatedly display on subplxel addressable display~15407 

-any scaled, subpixei optimized video frame followed by a sequence of 
one or more scaled subpixei optimized change bitmaps over the bitmap 
of that fiame at con^spondlng scaled positions on the frame-15408 
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-moving images with fixed subpixelation-ISSOO 

-store subplxel-optimlzed bitmap of image^15502 

-for each successive frame time-15503 

-calculate movement of image at fixed size and orientation, rounding 
location to nearest horizontal and vertical whole pixel location'-15504 
-display image at that location-'15506 

FIG. 155 



moving image with changing subpixelation-15600 
-store high resolution source image~15602 
-for each successive frame tfme-15603 

-calculate translation, rotation, and/ortransfomriation of high resolution 

source^'! 5604 

-generate scale-down and subpixel optimized bitmap of image with 
images mapping into subpixel grid associated with bitmap being a 
function of its tranlation, rotation, and/or transformation-^ 5606 
-display resulting subpixel optimized bitmap on subpixel display-ISBOS 

FIG. 156 



-subpixel optimazation of DVD video-'15700 

-decompress DVD video to a resolution higher than that at which it is to be 
displayed In subpixel optimized image-15702 

-scale and supixel optimize decompressed bitmaps of video images--15704 
-display scaled subpixel optimized bitmaps of video images on subpixel 
addressable display-15706 

FIG. 157 



-subpixel optimization of HDTV-15800 

-decompress HDTV video to a resolution higher than that at which it is to be 
displayed in subpixel optimized image'-15802 

-scale and supixel optimize decompressed bitmaps of video images-15804 
-display scaled subpixel optimized bitmaps of video images on subpbcel 
addressable display--15806 
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-subpixel optmization of mpeg4~15900 

-receive and decompress mpeg4 video-15902 

-use bicolor subpixel optimization with non-linear color balance in scaling 

down of bicolor objects'-15904 

-use multi-color subpixel optimization in scaling down of non-bicolor 
objectS'-ISQOe 

-display combination of bicolor and multicolor objects on subpixel optimized 
display moving subpixel optimized objects relative to screen-'15908 



-server subpixel optimization of scaled down, downloaded video-'16000 
-receives request for video and specification of subpixel display 
resolution7l6002 

-receives requested video content-16004 

-scales down and subpixel optimizes the received video .to subpixel resolution 

association with request-'16006 

-compresses video-16008 

-downloads it to requesting device-16010 



-proxy subpixel optimization of scaled down, downloaded video-16100 

-proxy computer code^-l 61 00 

-when receive request for video (and specification of subpixel 
resolutionh16102 



server~16103 
-when receive requested video 00^0^^-16104 

-scale down and subpixel optimizes video (to specified subpixel 
resolution)-16106 

-compress subpixel optimized video'-16108 
-download it to thin client--161 10 

-thin client code-1 61 12 

-in response to user input, send request for video to proxy (including 
subpixel resolution at which video is to be displayed)'-16113 
-when receive requested video from proxy-lol 14 
-decompress video-1 61 1 5 

-display scaled-down decompressed video on subpixel 
addressable display'^16116 
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-Electronic Ink code'-16200 

-If user enters electronic ink input-16202 

-record strokes as series of points and curves or lines in 
between-16204 

-draw ink on screen using subpixel optimization of lines and curves with 

non-linear color balance-1 6206 
-if user selects to scale up representation of electronic ink-'16208 

-produce subpixel optimized bitmap of ink's lines and curves using 

bicolor subpixel optimization with non-linear color balancing at the 

selected scaled up size"-'l6210 

-display scaled up image-1 6212 
-If user selects to scale down representation of electronic ink~16214 

-to produce subpixel optimized bitmap of ink's lines and curves using 

bicolor subpixel optimization with non-linear color balancing at the 

selected scaled down size-16216 

-display scaled down image'-16218 
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-subpixel optimized font bitmaps witli non-linear color balance--6000A 

-determine tightest rectangular array of rasterazition units into wliich cliaracter 
font shape fits, taking into account alignment of character-font shape relative 
to raterization units due to hinting18202 
-for each pixel row'-6002A 

-for each subpixel in a row-6004 

-determine a coverage value representing the percent of the 
subpixel which is covered by the font shape--6006 
-map resulting array of subpixel coverage values Into an an'ay of subpixel 
addressable pixels, aligning first column of rasterization units with leftmost 
subpixel of a pixel row-18204 

-pad an-ay with pixel column comprised of three subpixel's each to left-18206 
-pad an-ay with two, three, or four more sub-pbcel columns to right, so as to 
cause the total number of sub-pixel columns to be an even multiple of 
three--18208 

-adjust left and right side bearing values to compensate for padding subpbcel 
column on left and right side of bitmap-'18210 
-perform non-linear color balancing--^8212 
-convert to packed color value pixel bitmap--18214 

FIG. 182 



-drawing character string'-18300 

-set pen position to start position for string-18302 
-for each character of string to be displayed-9714A 

-access its associated font bitmap'-S^IS 

-set character start position to pen position'-18304 

-adjust pen position by left side bearing~18306 

-for each pixel value of the font bitmap'-971 8 
-if pixel is non-zero, draw pixel 18308 

-set pen position to character start position plush cunrent character's 

advance wldth'-1 831 0 



FIG. 183 
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-selected text reflow-18500 

-acxjess web page's contents-18502 

-perform first layout of web page's contents, placing text at different horizontal 
locations as indicated by web page'-18504 
-display elements of web page at positions determined by the first 
layoiit-1 8506 

-enable user to select a portion of text at a given horizontal location in display 
of first layout-18508 

-respond to user selection of such a portion of text by-l 8510 

-performing a second layout of the selected text by re-flowing it in a new 
column, at a different font size relative to new column's wldth'-18512 
-displaying layout of new column at scale that fills at least two thirds of 
width of screen-'ISSM 

FIG. 185 



-zoom to fit-1 8600 

-access web page's contents-18602 

-perform layout of web page's contents~18604 

-display all or portion of layout at first scale-^iseoe 

-enable user to drag pointing device across first scale layout display; -18608 

-If drag continues across screen boundary -18610 

scroll onto screen portions of layout at first scale previously off 

screen— 18612 
-if drag Is released-18614 

-define selected layout part based on position In first scale layout 

display of start and end of drag -1 861 6 
-display selected part of layout at second scale that fits selected layout part to 
screen. -18618 

FIG. 186 



-drag scroll -18700 

-access web page's contents-18702 
-perform layout of web page's contents-l 8704 
-display all or portion of layout'-18706 

-enable user to drag a pointing device across display of layout-18708 
-responding to any such drag across a boundary associated with a screen 
edge by scrolling onto screen, past the screen edge, portions of layout 
previously off screen-1 871 0 



FIG. 187 
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-clickzoofn^18800 

-access web page's contents'-'l8802 

-perform layout of web page's contents-18804 

-display all or portion of layout at first scale-'ISSOe 

-enable user to click a pointing device at a selected location in display of 

layout at first scale~18808 

-responding to such a click by perfonning a zoomed display of portion of layout 
around selected location--18810 

FIG. 188 

-zoomclick--18900 

-access web page's contents-18902 

-perform layout of web page's contents-18904 

-display all or portion of layout at first scale on touch screen~18906 

-If user presses touch screen at first position in first scale layout 

display-18908 

-replace first scale layout display with a display at larger scale of portion 

of layout that includes first position at substantially same position on 

screen as in first scale layout display-^-'JSQIO 

-display cursor slightly above location of touch-'18912 

-respond to any movement of touch by correspondingly moving cursor 

on display at second 8cale^18914 

-respond to any movement of touch across a boundary associated with 
a screen edge by scrolling onto screen, past the screen edge, portions 
of layout at second scale previously off screen-^ISQie 
-if user subsequently releases of press at a selected position in second 
scale layout display-18918 

-act as if a mouse click had occun^ed at corresponding position 

relative to web page'-18920 

-replace second scale layout display with first scale layout 
display-18922 



FIG. 189 
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-zoom out with greeking'-19000 

-access web page's contents'-19002 

-perform layout of web page's contentS'-19004 

-if user has selected given larger display scale'-'l9006 

-display portion of web page's layout at larger scale. including-^ISOOS 
-representing layout's images with bitmap images scaled for 
display at larger scale^l 901 0 

-representing layout's strings by bitmap images composed of 
font bitmaps sized for display at larger scale'-19012 
-if user has selected given smaller display scale-*! 9014 

-display portion of web page's layout at smaller scale, including-'19016 
-representing layout's images with bitmap images scaled for 
display at smaller scale--19018 

-representing layout's strings by bitmaps composed of a greeked 
text representation designed to indicate size and location of each 
such string at smaller scale-19020 



FIG. 190 
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